Human-object interaction action recognition method based on recombination sample learning

Abstract

The application provides a human-object interaction action recognition method based on recombination sample learning, which has the following characteristics: step S1, feature extraction and coding are performed on the to-be-recognized image to obtain a global feature vector and a position coding vector; step S2, the global feature vector, the position coding vector and a query vector group are input into a human-object pair decoder to obtain a human-object pair feature vector group; step S3, the global feature vector, the position coding vector and the human-object pair feature vector group are input into an interaction action decoder to obtain an interaction action feature vector group; step S4, the human-object pair feature vector group and the interaction action feature vector group are input into a feedforward neural network to obtain human-object pair prediction and interaction action prediction; and step S5, according to the human-object pair prediction, the interaction action prediction and N i human-object interaction, a recognition result is obtained. In summary, the method can improve the accuracy of human-object interaction action recognition.

Description

Technical Field

[0001] This invention relates to the field of human-object interaction action recognition, and specifically to a human-object interaction action recognition method based on recombined sample learning. Background Technology

[0002] As a crucial area of ​​artificial intelligence, action recognition has consistently been a research hotspot in both academia and industry. Human-object interaction action recognition is a key subtask within this field, requiring the location of interacting human-object instances within images and the identification of the interaction relationships between them. Human-object interaction action recognition is a vital core technology for deepening scene understanding and visual cognition, with broad application prospects and significant market demand in areas such as security monitoring, video retrieval, and autonomous driving.

[0003] In recent years, deep learning has become the mainstream method in the field of human-object interaction action recognition. With the Transformer architecture bringing about tremendous changes in the field of computer vision, recent research has proposed a variety of one-stage methods based on Transformer to achieve end-to-end human-object interaction action recognition.

[0004] Human-object interaction action recognition suffers from a severe long-tail data distribution problem, resulting in a highly uneven distribution of action categories. To address this issue, some existing studies have proposed few-shot or zero-shot human-object interaction action recognition methods, generalizing and transferring knowledge of head action categories to tail action categories or new action categories that have never been learned before. One type of method generates new training samples for the model to learn by recombining the features of different categories of human-object interactions.

[0005] However, existing methods of this kind generally adopt the traditional two-stage framework based on CNN. The visual features of human-object interaction generally lack global contextual information, and the reorganization of feature samples often leads to further loss of global contextual information, resulting in weaker representation ability and worse learning effect. Consequently, the recognition accuracy of human-object interaction action recognition models based on feature sample reorganization is poor. Summary of the Invention

[0006] This invention is made to solve the above-mentioned problems, and aims to provide a method for recognizing human-object interaction actions based on recombined sample learning.

[0007] This invention provides a method for recognizing human-object interaction actions based on recombined sample learning, used to identify actions based on N... q A set of query vectors Q with randomly initialized query vectors p and N iHuman-like interaction for recognizing images and obtaining recognition results has the following characteristics and includes the following steps: Step S1, input the image to be recognized into a convolutional neural network for feature extraction, and then encode it through a Transformer encoder to obtain a global feature vector X. s And the location encoding vector E; step S2, the global feature vector X s Location encoding vector E and query vector group Q p Input the person-object pair decoder to get N q The feature vectors of the person-object pair are used as the feature vector set R of the person-object pair. p Step S3, convert the global feature vector X s Location encoding vector E and person-object pair feature vector group R p Input the interactive action decoder to get N q The interaction action feature vectors of the person-object pair are used as the interaction action feature vector set R. i Step S4, convert the person-object pair feature vector group R p and interactive action feature vector group R i Inputting into a feedforward neural network yields human-object pair predictions. And interactive action prediction Y i ={a n n∈{1,2,...,N q Step S5: Predict Y based on person-object pairs. p Interactive action prediction Y i and N i Human-like interaction, resulting in N q Person-object pair person-object interaction prediction results n∈{1,2,...,N q As the recognition result, among which, Let the bounding box of the nth person-person pair be an example of a human body. Let o be the object border of the nth person-object pair. n Let a be the object category confidence vector containing the predicted confidence of all object categories for the nth person-object pair. n Let y' be the action category confidence vector for the nth person-person pair, containing all action classes. n For the prediction result of the person-object interaction of the nth person-object pair, u n Let v be the predicted object category for the nth person-object pair. n For the predicted action category of the nth person-object pair, Let be the maximum confidence level of the nth person-object pair.

[0008] The human-object interaction action recognition method based on recombined sample learning provided by this invention may also have the following feature: wherein step S1 includes the following sub-steps: step S1-1, inputting the image to be recognized into a convolutional neural network for feature extraction to obtain a visual feature map X. v Step S1-2, based on visual feature map X v The position encoding vector E is obtained; in steps S1-3, the visual feature map X is... v The positional encoding vector E is input into the Transformer encoder for encoding, resulting in the global feature vector X. s .

[0009] The human-object interaction action recognition method based on recombined sample learning provided by this invention may also have the following features: wherein the feedforward neural network includes a human-object classification feedforward network and an interaction action classification feedforward network, and step S4 includes the following sub-steps: Step S4-1, grouping the interaction action feature vectors R i Inputting the human-object classification feedforward network yields N. q Person-object pair prediction; Step S4-2, group the person-object pair feature vectors R p and interactive action feature vector group R i The input is then concatenated and fed forward into an interactive action classification network to obtain N. q Predicting the interaction actions of person-object pairs.

[0010] The human-object interaction action recognition method based on recombined sample learning provided by this invention may also have the following feature: wherein step S5 includes the following sub-steps: step S5-1, for each human-object pair, according to N i Human-object interaction involves multiplying the action category confidence vector and the object category confidence vector to obtain N. i Step S5-2: For each person-object pair, select the product of the highest confidence scores as the maximum confidence score of the person-object pair, and use the object category and action category corresponding to the maximum confidence score as the predicted object category and predicted action category of the person-object pair, respectively; Step S5-3: For N... q Sort the person-object pairs by their maximum confidence scores from highest to lowest to obtain N. q Prediction results of person-object interactions between individuals and objects.

[0011] The human-object interaction action recognition method based on recombined sample learning provided by this invention may also have the following feature: wherein, through training samples containing multiple training images and their corresponding real labels, N i Human-like interaction and containing N q A set of query vectors Q with randomly initialized query vectors pTraining a model including a convolutional neural network, a Transformer encoder, a human-object pair decoder, an interactive action decoder, and a feedforward neural network involves the following steps: Step T1, randomly select training images I1 and I2 from the training samples, input training images I1 and I2 into the convolutional neural network for feature extraction, and then encode them through the Transformer encoder to obtain the global feature vector and positional encoding vector of the training image, respectively; Step T2, for each training image, group the global feature vector, positional encoding vector, and query vector Q. p Input the person-object pair decoder to get N q The feature vectors of the person-object pair are used as the person-object pair feature vector group; in step T3, for each training image, the global feature vector, the position encoding vector, and the person-object pair feature vector group are input into the interactive action decoder to obtain N. q The interaction action feature vectors of the person-object pair are used as the interaction action feature vector group; Step T4, for each training image, the person-object pair feature vector group and the interaction action feature vector group are input into the feedforward neural network to obtain the person-object pair prediction and the interaction action prediction. The person-object pair prediction and the interaction action prediction are directly combined to obtain the person-object interaction prediction; Step T5, for each training image, the person-object interaction prediction is matched with the corresponding real label of the training image according to the Hungarian algorithm to obtain the most accurate person-object interaction prediction; Step T6, based on the most accurate person-object interaction prediction of the two training images, the corresponding most accurate person-object pair feature vector and the most accurate interaction action feature vector are obtained. Step T1 involves cross-recombining or internally recombining the interaction action feature vectors to obtain a recombined human-object interaction feature vector group. This group is then input into the interaction action classification feedforward network to obtain a recombined human-object interaction prediction. The ground truth labels corresponding to the two training images are then recombined to obtain recombined ground truth labels. Step T7 involves calculating a loss function based on the most accurate human-object interaction predictions and ground truth labels for the two training images, as well as the recombined human-object interaction predictions and recombined ground truth labels. The parameters of the interaction action decoder are then optimized based on the loss function calculation results. Step T8 involves repeating steps T1 to T7 until all training images in the training samples have optimized the model parameters. The model training is then complete, and the most accurate human-object interaction prediction for the k-th training image is obtained. For the most accurate prediction of the person-object interaction for the nth person-object pair in the kth training image, For the most accurate human bounding box of the nth person-person pair in the kth training image, For the most accurate object bounding box of the nth person-object pair in the kth training image, o kn Let a be the most accurate object category confidence vector for the nth person-object pair in the kth training image. knFor the nth person-object pair in the kth training image, the most accurate action category confidence vector is... denoted as the true number of human-object interaction instances in the k-th training image.

[0012] The human-object interaction action recognition method based on recombined sample learning provided by this invention may also have the following feature: when performing cross-recombination based on the most accurate human-object interaction predictions of training image I1 and training image I2, step T6 includes the following sub-steps: Step T6-1, based on the most accurate human-object interaction predictions of training image I1... Obtain the most accurate person-object pair feature vector set of training image I1 and the most accurate interactive action feature vector set Step T6-2: Based on the most accurate human-object interaction prediction of training image I2 Obtain the most accurate person-object pair feature vector set of training image I2 and the most accurate interactive action feature vector set Step T6-3: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one to obtain the recombined character interaction feature group. Step T6-4: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one to obtain the recombined character interaction feature group. Step T6-5 involves recombining the character interaction feature groups. and reorganized character interaction feature group The interaction action predictions are obtained by inputting each action into the interaction action classification feedforward network. and interactive action prediction Step T6-6: Predict Y from the person-object pairs in the training image I1. p1 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Predict the person-object pair from training image I2. p2 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Recombinant Human-Object Interaction Prediction Recombinant Human-Object Interaction Prediction For reconstructing human-object interaction prediction; Steps T6-7, based on the true labels corresponding to the training image I1 Get real-life labels for people and things and interactive action real tags Based on the true label corresponding to the training image I2 Get real-life labels for people and things and interactive action real tags Step T6-8, according to N i Human-like interaction, connecting people and objects with real-world labels. and interactive action real tags By pairing and combining them one by one, corresponding recombinant human-object interaction predictions can be obtained. Recombination True Label To assign real labels to people and things and interactive action real tags By pairing and combining them one by one, corresponding recombinant human-object interaction predictions can be obtained. Recombination True Label Recombining Real Labels and reorganization of real labels To reconstruct the true labels.

[0013] The human-object interaction action recognition method based on recombined sample learning provided by the present invention may also have the following feature: wherein, when performing internal recombination based on the most accurate human-object interaction prediction of training image I1 and training image I2, step T6 includes the following sub-step: Step T6-1, based on the most accurate human-object interaction prediction of training image I1 Obtain the most accurate person-object pair feature vector set of training image I1 and the most accurate interactive action feature vector set Step T6-2: Based on the most accurate human-object interaction prediction of training image I2 Obtain the most accurate person-object pair feature vector set of training image I2 and the most accurate interactive action feature vector set Step T6-3: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one, and then the original combination of person-object pair feature vectors and interaction action feature vectors is removed to obtain the recombined person interaction feature group. Step T6-4: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one, and then the original combination of person-object pair feature vectors and interaction action feature vectors is removed to obtain the recombined person interaction feature group. Step T6-5 involves recombining the character interaction feature groups. and reorganized character interaction feature group The interaction action predictions are obtained by inputting each action into the interaction action classification feedforward network. and interactive action prediction Step T6-6: Predict Y from the person-object pairs in the training image I1. p1 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Predict the person-object pair from training image I2. p2 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Recombinant Human-Object Interaction Prediction Recombinant Human-Object Interaction Prediction For reconstructing human-object interaction prediction; Steps T6-7, based on the true labels corresponding to the training image I1 Get real-life labels for people and things and interactive action real tags Based on the true label corresponding to the training image I2 Get real-life labels for people and things and interactive action real tags Step T6-8, according to N i Human-like interaction, connecting people and objects with real-world labels. and interactive action real tags Pair them one by one, then remove the true labels corresponding to the training image I1. Obtain the corresponding recombinant human-object interaction prediction Recombination True Label To assign real labels to people and things and interactive action real tags Pair them one by one, then remove the true labels corresponding to the training image I2. Obtain the corresponding recombinant human-object interaction prediction Recombination True Label Recombining Real Labels and reorganization of real labels To reconstruct the true labels.

[0014] The human-object interaction action recognition method based on recombined sample learning provided by this invention can also have the following feature: wherein the combination of an object category in the real label of the human-object interaction with an action category in the interaction action feature vector of the real label of the interaction action exceeds N. iIn human-object interaction, during the reconstructing of real labels, the element value corresponding to the object category in the interaction action feature vector is set to 0. The combination of a human-object interaction with an object category in the real label and all action categories in the interaction action feature vector of the real label exceeds N. i In human-object interaction, the combination of object category and interaction action feature vector is removed during the reconstruction of real labels. The combination of one object category in the real label and all action categories in all interaction action feature vectors of the real label exceeds N. i In human-object interaction, the object category in the reconstructed real label corresponds to an interaction action feature vector with all zeros.

[0015] The human-object interaction action recognition method based on recombined sample learning provided by this invention may also have the following feature: wherein, in step T7, the formula for calculating the loss function is as follows: L=λ b ·L b +λ u ·L u +λ o ·L o +λ a ·L a In the formula L b L u L o and L a These are the bounding box regression loss function, bounding box intersection-over-union (IoU) loss function, object class loss function, and action class loss function, respectively, λ. b , λ u , λ o and λ a Here, L represents the weight hyperparameters of the bounding box regression loss function, the bounding box intersection-over-union (OCU) loss function, the object category loss function, and the action category loss function, respectively. The formula for calculating the loss function is as follows: L batch =ρ·L orig +(1-ρ)·L compo In the formula L batch L is the result of the loss function calculation. orig The result of calculating the loss function L for the most accurate prediction of human-object interaction and the true label is L. compo The result of calculating the loss function L for recombining human-object interaction prediction and recombining real labels is given, where ρ is the weight hyperparameter that balances the two.

[0016] The role and effect of invention

[0017] The human-object interaction action recognition method based on recombined sample learning according to the present invention enhances the representational ability of training images without relying on additional information by cross-recombining or internal recombining during model training, thus realizing cross-class transfer and generalization of human-object interaction action knowledge. Therefore, the human-object interaction action recognition method based on recombined sample learning of the present invention can alleviate the performance imbalance caused by long-tail distribution and improve the accuracy of human-object interaction action recognition. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the human-object interaction action recognition method based on recombined sample learning in an embodiment of the present invention;

[0019] Figure 2 This is a schematic diagram of the training model process in an embodiment of the present invention;

[0020] Figure 3 This is a schematic diagram of the cross-recombination process in an embodiment of the present invention;

[0021] Figure 4 This is a schematic diagram illustrating the principle of cross-recombination in an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of the internal reorganization process in an embodiment of the present invention;

[0023] Figure 6 This is a schematic diagram illustrating the principle of internal reorganization in an embodiment of the present invention. Detailed Implementation

[0024] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the human-object interaction action recognition method based on recombined sample learning of this invention.

[0025] Figure 1 This is a flowchart illustrating the human-object interaction action recognition method based on recombined sample learning in an embodiment of the present invention.

[0026] like Figure 1 As shown in this embodiment, the human-object interaction action recognition method based on recombined sample learning is used to recognize actions based on N... q A set of query vectors Q with randomly initialized query vectors p and N i Human-like interaction is used to identify images and obtain recognition results, including the following steps:

[0027] Step S1: Input the image to be recognized into a convolutional neural network for feature extraction, and then encode it through a Transformer encoder to obtain a global feature vector X. sAnd the position encoding vector E.

[0028] Step S1 includes the following sub-steps:

[0029] Step S1-1: Input the image to be recognized into a convolutional neural network for feature extraction to obtain a visual feature map X. v The dimensions are H×W×C, where H, W, and C represent the height, width, and global feature map dimensions, respectively. The first two dimensions are then merged, resulting in (H×W)×C. In this embodiment, the convolutional neural network is a ResNet-50 network pre-trained on the COCO dataset and fine-tuned on the HICO-DET dataset. Convolutional blocks 1 through 4 are used for feature extraction, and the global feature dimension C is set to 256.

[0030] Step S1-2, based on visual feature map X v This yields a position encoding vector E, which is used to record the visual feature map X. v The position information of each pixel in the vector is also in the form of (H×W)×C. In this embodiment, the position encoding vector E is obtained according to the sine and cosine position encoding calculation method given in DETR.

[0031] Steps S1-3: Transfer the visual feature map X v The positional encoding vector E is input to the Transformer encoder for encoding, and the global context is aggregated through a self-attention mechanism to obtain the global feature vector X. s Global feature vector X s The dimension is also (H×W)×C.

[0032] Step S2, convert the global feature vector X s Location encoding vector E and query vector group Q p The input is the human-object pair decoder, which will process the global feature vector X. s Contextual information about people and objects is aggregated into the query vector to obtain N. q The feature vectors of the person-object pair are used as the feature vector set R of the person-object pair. p Human-object pair feature vector set R p and query vector group Q p The dimensions are all N q ×C q C q This represents the feature dimension of the query vector. In this embodiment, the feature dimension C is... q It is 256.

[0033] Step S3, convert the global feature vector X s Location encoding vector E and person-object pair feature vector group R pThe input is the interactive action decoder, which will process the global feature vector X. s Contextual information about interactions between people and objects is aggregated into the query vector to obtain N. q The interaction action feature vectors of the person-object pair are used as the interaction action feature vector set R. i Interactive action feature vector group R i The dimension is N q ×C q .

[0034] Step S4, convert the human-object pair feature vector group R p and interactive action feature vector group R i Inputting into a feedforward neural network yields human-object pair predictions. n∈{1,2,...,N q}} and interactive action prediction Y i ={a n n∈{1,2,...,N q}}.

[0035] The feedforward neural network includes a human-object classification feedforward network and an interactive action classification feedforward network. Step S4 includes the following sub-steps:

[0036] Step S4-1, group the interactive action feature vectors R i Inputting the human-object classification feedforward network yields N. q Person-object pair prediction n∈{1,2,...,N q}}.

[0037] Step S4-2, convert the human-object pair feature vector group R p and interactive action feature vector group R i The input is then concatenated and fed forward into an interactive action classification network to obtain N. q Prediction of interaction actions Y between person and object i ={a n n∈{1,2,...,N q In this embodiment, the spliced ​​result forms a dimension of N. q ×(C q ×2) human-object interaction feature vector

[0038] in, Let the bounding box of the nth person-person pair be an example of a human body. and These are the x and y coordinates of the top-left corner of the human body's outline, respectively. and These are the x and y coordinates of the bottom right corner of the human body's border, respectively. Let the object border of the nth person-object pair be... and These are the x and y coordinates of the top-left corner of the object's border, respectively. and These are the x and y coordinates of the bottom right corner of the object's border, respectively. n Let be the object category confidence vector containing the predicted confidence of all object categories for the nth person-object pair. N represents the predicted confidence of the u-th object category in the object category confidence vector. o a is the total number of object categories. n Let be the action category confidence vector for the nth person-object pair, containing all action classes. N represents the predicted confidence of the v-th action category in the confidence vector of this action category. a This represents the total number of action categories.

[0039] Step S5, predict Y based on person-object pairs p Interactive action prediction Y i and N i Human-like interaction, resulting in N q Person-object pair person-object interaction prediction results n∈{1,2,...,N q As the recognition result, y' n For the prediction result of the person-object interaction of the nth person-object pair, u n Let v be the predicted object category for the nth person-object pair. n For the predicted action category of the nth person-object pair, Let be the maximum confidence level of the nth person-object pair.

[0040] Step S5 includes the following sub-steps:

[0041] Step S5-1, for each person-person pair, according to N i Human-object interaction involves multiplying the action category confidence vector and the object category confidence vector to obtain N. i The product of confidence levels.

[0042] Confidence product c w The calculation expression is as follows:

[0043]

[0044] In the formula Let be the prediction confidence score for the object category of the w-th person-object interaction. Let w be the prediction confidence level for the action category of the w-th person-object interaction.

[0045] Step S5-2: For each person-object pair, select the product of the highest confidence scores as the maximum confidence score of the person-object pair, and use the object category and action category corresponding to the maximum confidence score as the predicted object category and predicted action category of the person-object pair, respectively.

[0046] Step S5-3, for N q Sort the person-object pairs by their maximum confidence scores from highest to lowest to obtain N. q Prediction results of person-object interactions between individuals and objects.

[0047] Figure 2 This is a schematic diagram of the training model process in an embodiment of the present invention.

[0048] like Figure 2 As shown, training samples containing multiple training images and their corresponding real labels, N i Human-like interaction and containing N q A set of query vectors Q with randomly initialized query vectors p Training a model that includes convolutional neural networks, Transformer encoders, human-object pair decoders, interactive action decoders, and feedforward neural networks involves the following steps:

[0049] Step T1: Randomly select a pair of training images I from the training samples. k k = 1,2, which are training images I1 and I2. The training images I1 and I2 are input into the convolutional neural network for feature extraction, and then encoded by the Transformer encoder to obtain the global feature vector and position encoding vector of the training image, respectively.

[0050] Step T2: For each training image, combine the global feature vector, location encoding vector, and query vector into Q. p Input the person-object pair decoder to get N q The feature vectors of the person-object pair are used as the feature vector set R of the person-object pair. pk That is, the feature vector set R corresponding to the person-object pair of the training image I1 p1 The training image I2 corresponds to the human-object pair feature vector group R. p2 R p1 and R p2 The dimensions are all N q ×C q .

[0051] Step T3: For each training image, input the global feature vector, location encoding vector, and person-object pair feature vector group into the interactive action decoder to obtain N. q The interaction action feature vectors of the person-object pair are used as the interaction action feature vector set R. ik That is, the feature vector group R corresponding to the interactive action of the training image I1 i1 The training image I2 corresponds to the interactive action feature vector group R. i2 R i1 and R i2 The dimensions are all N q ×C q .

[0052] Step T4: For each training image, input the human-object pair feature vector set and the interaction action feature vector set into the feedforward neural network to obtain the human-object pair prediction. n∈{1,2,…,N q}} and interactive action prediction Y ik ={a kn n∈{1,2,…,N q By directly combining human-object pair prediction and interaction action prediction, we obtain human-object interaction prediction. n∈{1,2,…,N q}}, For the human bounding box of the nth person-person pair in the kth training image, For the object bounding box of the nth person-object pair in the kth training image, o kn Let a be the object category confidence vector of the nth person-object pair in the kth training image. kn This is the action category confidence vector for the nth person-object pair in the kth training image.

[0053] Step T5: For each training image, the human-object interaction prediction is matched with the corresponding real label of the training image according to the Hungarian algorithm to obtain the most accurate human-object interaction prediction.

[0054] The matching process involves using the Hungarian algorithm to match the real labels. And predict Y k One-to-one matching, calculating the total matching cost for each real tag. Choose the prediction result with the lowest total matching cost. The set of all the most accurate predictions, corresponding to the real labels, is the most accurate prediction of human-object interaction.

[0055] The matching cost follows the calculation method of the existing work QPIC, which is the total matching cost H of predicting the human-object interaction between the i-th ground truth label and the j-th human-object pair in the training image. i,j for:

[0056]

[0057] in and Let α represent the costs of bounding box regression, bounding box intersection-over-union (IoU), object category matching, and action category matching, respectively. b α u α o and α a These represent the weight hyperparameters of each matching cost, which are set to 2.5, 1, 1, and 1 respectively in this embodiment.

[0058] Among them, training image I k Corresponding real tags For the human body outline, The object's border. This is a vector of object category labels. This is an action category label vector, where the element corresponding to the category has a value of 1, and the rest have a value of 0.

[0059] The most accurate human-object interaction prediction for the kth training image For the most accurate prediction of the person-object interaction for the nth person-object pair in the kth training image, For the most accurate human bounding box of the nth person-person pair in the kth training image, For the most accurate object bounding box of the nth person-object pair in the kth training image, o kn Let α be the most accurate object category confidence vector for the nth person-object pair in the kth training image. kn For the nth person-object pair in the kth training image, the most accurate action category confidence vector is... denoted as the true number of human-object interaction instances in the k-th training image.

[0060] Step T6: Based on the most accurate human-object interaction predictions from the two training images, obtain the corresponding most accurate human-object pair feature vector and most accurate interaction action feature vector. Perform cross-recombination or internal recombination on the most accurate human-object pair feature vector and the most accurate interaction action feature vector to obtain a recombined human-object interaction feature vector group. Input this group into the interaction action classification feedforward network to obtain the recombined human-object interaction prediction. Then, recombinate the corresponding ground truth labels of the two training images to obtain the recombined ground truth labels. In this embodiment, to balance computational overhead and training effect, the total number of recombined feature vectors for the recombined human-object interaction prediction of a training image does not exceed 200, of which the number of recombined feature vectors from cross-recombination does not exceed 100, and the number of recombined feature vectors from internal recombination does not exceed 100. Feature vectors exceeding these limits are directly discarded.

[0061] Figure 3 This is a schematic diagram of the cross-recombination process in an embodiment of the present invention.

[0062] like Figure 3 As shown, when performing cross-recombination based on the most accurate human-object interaction predictions of training images I1 and I2, step T6 includes the following sub-steps:

[0063] Step T6-1: Based on the most accurate human-object interaction prediction of training image I1 Obtain the most accurate person-object pair feature vector set of training image I1 and the most accurate interactive action feature vector set and All dimensions are

[0064] Step T6-2: Based on the most accurate human-object interaction prediction of training image I2 Obtain the most accurate person-object pair feature vector set of training image I2 and the most accurate interactive action feature vector set and All dimensions are

[0065] Step T6-3: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one to obtain the recombined character interaction feature group. Reorganize the character interaction feature group The dimension is

[0066] Step T6-4: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one to obtain the recombined character interaction feature group. Reorganize the character interaction feature group The dimension is

[0067] Step T6-5 involves recombining the character interaction feature groups. and reorganized character interaction feature group The interaction action predictions are obtained by inputting each action into the interaction action classification feedforward network. and interactive action prediction

[0068] Step T6-6: Predict Y from the person-object pairs in the training image I1. p1 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Predict the person-object pair from training image I2. p2 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Recombinant Human-Object Interaction Prediction Recombinant Human-Object Interaction Prediction To predict human-object interactions in reorganization.

[0069] Steps T6-7: Based on the true labels corresponding to the training image I1 Get real-life labels for people and things and interactive action real tags Based on the true label corresponding to the training image I2 Get real-life labels for people and things and interactive action real tags

[0070] Step T6-8, according to N i Human-like interaction, connecting people and objects with real-world labels. and interactive action real tags By pairing and combining them one by one, corresponding recombinant human-object interaction predictions can be obtained. Recombination True Label To assign real labels to people and things and interactive action real tags By pairing and combining them one by one, corresponding recombinant human-object interaction predictions can be obtained. Recombination True Label Recombining Real Labels and reorganization of real labels To reconstruct the true labels.

[0071] Figure 4 This is a schematic diagram illustrating the principle of cross-recombination in an embodiment of the present invention.

[0072] like Figure 4 As shown, the first training image contains two person-object interaction actions, namely "person sit on couch" and "person readbook". The second training image contains two person-object interaction actions, namely "person carry suitcase" and "person wear backpack". From the first training image, the two person-object pair feature vectors P1-1 and P1-2 of the most accurate person-object pair feature vector group and the two interaction action feature vectors I1-1 and I1-2 of the most accurate interaction action feature vector group are obtained. From the second training image, the two person-object pair feature vectors P2-1 and P2-2 of the most accurate person-object pair feature vector group and the two interaction action feature vectors I2-1 and I2-2 of the most accurate interaction action feature vector group are obtained. By concatenating the person-object pair feature vectors P1-1 and P1-2 one by one with the interaction action feature vectors I2-1 and I2-2, we obtain the recombined person interaction feature group R1, including: P1-1 and I2-1, P1-1 and I2-2, P1-2 and I2-1, and P1-2 and I2-2. The corresponding real labels are then recombined to obtain the corresponding "carry couch", "wear couch", "carrybook", and "wearbook". After filtering, the recombined real label "carrybook" is obtained. Similarly, by concatenating the person-object pair feature vectors P2-1 and P2-2 one by one with the interaction action feature vectors I1-1 and I1-2, we obtain the recombined person interaction feature group R2, including: P2-1 and I1-1, P2-1 and I1-2, P2-2 and I1-1, and P2-2 and I1-2. The corresponding real labels are then recombined to obtain the corresponding "sit on suitcase", "read suitcase", "siton backpack", and "readbackpack". After filtering, no suitable recombined real labels are found.

[0073] Figure 5 This is a schematic diagram of the internal reorganization process in an embodiment of the present invention.

[0074] like Figure 5 As shown, when performing internal reconstruction based on the most accurate human-object interaction predictions of training images I1 and I2, step T6 includes the following sub-steps:

[0075] Step T6-1: Based on the most accurate human-object interaction prediction of training image I1 Obtain the most accurate person-object pair feature vector set of training image I1 and the most accurate interactive action feature vector set

[0076] Step T6-2: Based on the most accurate human-object interaction prediction of training image I2 Obtain the most accurate person-object pair feature vector set of training image I2 and the most accurate interactive action feature vector set

[0077] Step T6-3: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one, and then the original combination of person-object pair feature vectors and interaction action feature vectors is removed to obtain the recombined person interaction feature group.

[0078] Step T6-4: The most accurate person-object pair feature vector group of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one, and then the original combination of person-object pair feature vectors and interaction action feature vectors is removed to obtain the recombined person interaction feature group.

[0079] Step T6-5 involves recombining the character interaction feature groups. and reorganized character interaction feature group The interaction action predictions are obtained by inputting each action into the interaction action classification feedforward network. and interactive action prediction

[0080] Step T6-6: Predict Y from the person-object pairs in the training image I1. p1 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Predict the person-object pair from training image I2. p2 and interactive action prediction Combining these elements yields a recombinant human-object interaction prediction. Recombinant Human-Object Interaction Prediction Recombinant Human-Object Interaction Prediction To predict human-object interactions in reorganization.

[0081] Steps T6-7: Based on the true labels corresponding to the training image I1 Get real-life labels for people and things and interactive action real tags Based on the true label corresponding to the training image I2 Get real-life labels for people and things and interactive action real tags

[0082] Step T6-8, according to N i Human-like interaction, connecting people and objects with real-world labels. and interactive action real tags Pair them one by one, then remove the true labels corresponding to the training image I1. Obtain the corresponding recombinant human-object interaction prediction Recombination True Label To assign real labels to people and things and interactive action real tags Pair them one by one, then remove the true labels corresponding to the training image I2. Obtain the corresponding recombinant human-object interaction prediction Recombination True Label Recombining Real Labels and reorganization of real labels To reconstruct the true labels.

[0083] Among them, the combination of an object category in the real-world label of a person-object pair and an action category in the feature vector of an interactive action in the real-world label of an interactive action exceeds N. i When human-object interaction occurs, the element value corresponding to the object category in the interaction action feature vector is set to 0 in the reconstructed real label.

[0084] The combination of all action categories in the feature vector of a person-object pair with a real-world label object category and an interactive action with a real-world label interactive action exceeds N. i When human-object interaction occurs, the combination of object category and interaction action feature vector is removed during the reconstruction of real labels.

[0085] The combination of a person-object pair with an object category in the real-world label and all action categories in the feature vectors of all interactive actions in the real-world label exceeds N. i In human-object interaction, the object category in the reconstructed real label corresponds to an interaction action feature vector with all zeros. Figure 6 This is a schematic diagram illustrating the principle of internal reorganization in an embodiment of the present invention.

[0086] like Figure 6As shown, the first training image contains two person-object interaction actions, namely "person sit on couch" and "person read book". The second training image contains two person-object interaction actions, namely "person carry suitcase" and "person wear backpack". From the first training image, the two person-object pair feature vectors P1-1 and P1-2 of the most accurate person-object pair feature vector group and the two interaction action feature vectors I1-1 and I1-2 of the most accurate interaction action feature vector group are obtained. From the second training image, the two person-object pair feature vectors P2-1 and P2-2 of the most accurate person-object pair feature vector group and the two interaction action feature vectors I2-1 and I2-2 of the most accurate interaction action feature vector group are obtained. The person-object pair feature vectors P1-1 and P1-2 are concatenated one by one with the interaction action feature vectors I1-1 and I1-2, and the original person interaction features are removed to obtain the recombined person interaction feature group R1, which includes: P1-1 and I1-2 and P1-2 and I1-1. The corresponding real labels are then recombined to obtain the corresponding "read couch" and "sit on book". After filtering, no suitable real labels are found for recombination. The person-object pair feature vectors P2-1 and P2-2 are concatenated one by one with the interaction action feature vectors I2-1 and I2-2, and the original person interaction features are removed to obtain the recombined person interaction feature group R2, which includes: P2-1 and I2-2 and P2-2 and I2-1. The corresponding real labels are then recombined to obtain the corresponding "wear suitcase" and "carry backpack". After filtering, the recombined real label "carry backpack" is obtained.

[0087] Step T7: Calculate the loss function based on the most accurate human-object interaction prediction and real label of the two training images, as well as the reconstructed human-object interaction prediction and reconstructed real label. Optimize the parameters of the interaction action decoder based on the loss function calculation results.

[0088] In step T7, the formula for calculating the loss function is as follows:

[0089] L=λ b ·L b +λ u ·L u +λ o ·L o +λ a ·L a ,

[0090] In the formula L b L u L o and L aThese are the bounding box regression loss function, bounding box intersection-over-union (IoU) loss function, object class loss function, and action class loss function, respectively, λ. b , λ u , λ o and λ a λ represents the weight hyperparameters of the bounding box regression loss function, the bounding box intersection-union loss function, the object category loss function, and the action category loss function, respectively. L is the loss function, and in this embodiment, λ is the weight hyperparameter. b , λ u , λ o and λ a Set them to 2.5, 1, 1 and 1 respectively.

[0091] The formula for calculating the loss function result is as follows:

[0092] L batch =ρ·L orig +(1-ρ)·L compo ,

[0093] In the formula L batch L is the result of the loss function calculation. orig The result of calculating the loss function L for the most accurate prediction of human-object interaction and the true label is L. compo The result of calculating the loss function L for recombining human-object interaction prediction and recombining real labels is given. ρ is the weight hyperparameter that balances the two. In this embodiment, ρ is set to 0.9.

[0094] Step T8: Repeat steps T1 to T7 until all training images in the training samples have optimized the model parameters, then the model training is complete.

[0095] In this embodiment, the training of the model constructed by the human-object interaction action recognition method based on recombined sample learning of the present invention follows the settings of the existing working CDN, using AdamW as the optimizer. The dataset is traversed a total of 100 times. The dataset includes multiple training images and corresponding ground truth labels. Each time, two training images and corresponding ground truth labels are randomly selected to train the model. If all training images in the dataset are used to train the model, then the dataset is traversed once. The learning rate for the first 60 iterations is set to 10. -4 The learning rate dropped to 10% from the 61st to the 90th iteration. -5 The 91st to 100th iterations are fine-tuning stages for the encoder and feedforward network. The method proposed in the existing work CDN for dynamically adjusting the weights of action categories in the loss function is adopted to finally obtain the trained model.

[0096] In this embodiment, the human-object interaction action recognition method, VCL method, FCL method, and ConsNet method based on recombined sample learning of the present invention are constructed under their respective convolutional neural networks to obtain the model of the present invention, the VCL model, the FCL model, and the ConsNet model. The human-object interaction action recognition of each model is tested on the public dataset V-COCO. The mAP index of each model is shown in the table below:

[0097]

[0098] The first column of the table lists the names of each model, the second column lists the convolutional neural networks used by each model, the third column lists whether each model introduces additional information, and the fourth column lists the mAP value corresponding to the recognition results of each model. For example, the cell in the fifth row and fourth column indicates that the mAP index (average precision) of the model in this invention for human-object interaction action recognition on the V-COCO dataset is 57.24%. As can be seen from the table above, compared with the other three models, the model in this invention not only does not introduce additional information, but also has the best human-object interaction action recognition prediction accuracy on the V-COCO dataset.

[0099] In this embodiment, the human-object interaction action recognition method based on recombined sample learning of the present invention, Analogy method, Functional method, VCL method, ATL method, FCL method, and ConsNet method are used to construct corresponding models under their respective convolutional neural networks, resulting in the model of the present invention, Analogy model, Functional model, VCL model, ATL model, FCL model, and ConsNet model. Human-object interaction action recognition of each model is tested on three test sets of the public dataset HICO-DET: all categories, rare categories, and non-rare categories. The mAP index of each model is shown in the table below:

[0100]

[0101] The first column of the table lists the names of each model, the second column lists the convolutional neural networks used by each model, the third column lists whether each model introduces additional information, the fourth column lists the mAP values ​​of each model's recognition results on the all-class test set, the fifth column lists the mAP values ​​of each model's recognition results on the rare-class test set, and the sixth column lists the mAP values ​​of each model's recognition results on the non-rare-class test set. For example, the cell in the eighth row and fourth column indicates that the mAP index (average precision) of the model in this invention for human-object interaction action recognition on the all-class test set of the HICO-DET dataset is 29.63%. As can be seen from the table above, compared with the other six models, the model in this invention not only does not introduce additional information, but also has the best human-object interaction action recognition prediction accuracy on all three test sets of the HICO-DET dataset.

[0102] The role and effect of the embodiments

[0103] According to the human-object interaction action recognition method based on recombined sample learning involved in this embodiment, cross-recombination or internal recombination is used during model training to enhance the representational ability of training images without relying on additional information, thereby realizing cross-class transfer and generalization of human-object interaction action knowledge. In summary, this method can alleviate the performance imbalance caused by long-tail distribution and improve the accuracy of human-object interaction action recognition.

[0104] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for recognizing human-object interactions based on recombined sample learning, used to recognize actions based on N... q A set of query vectors Q with randomly initialized query vectors p and N i A human-like interactive method is used to identify images to obtain recognition results, characterized in that... Includes the following steps: Step S1: Input the image to be recognized into a convolutional neural network for feature extraction, and then encode it through a Transformer encoder to obtain a global feature vector X. s and the position encoding vector E; Step S2, the global feature vector X s The location encoding vector E and the query vector group Q p Input the person-object pair decoder to get N q The feature vectors of the person-object pair are used as the feature vector set R of the person-object pair. p ; Step S3, the global feature vector X s The location encoding vector E and the person-object pair feature vector group R p Input interactive action decoder to obtain the N q The interaction action feature vectors of the aforementioned person-object pairs are used as the interaction action feature vector group R. i ; Step S4, convert the person-object pair feature vector group R p and the interactive action feature vector group R i Inputting into a feedforward neural network yields human-object pair predictions. And interactive action prediction Y i ={a n n∈{1,2,...,N q }}; Step S5, predict Y based on the person-object pair p The interactive action prediction Y i and the N i Human-like interaction, to obtain the N q Person-object pair person-object interaction prediction results As the identification result in, The bounding box of the nth person-object pair. For the object border of the nth person-object pair, o n Let a be the object category confidence vector containing the predicted confidence of all object categories for the nth person-object pair. n Let y′ be the action category confidence vector containing all action classes for the nth person-object pair. n For the prediction result of the human-object interaction of the nth human-object pair, u n For the predicted object category of the nth person-object pair, v n For the nth person-object pair, the predicted action category is... The maximum confidence level for the nth person-object pair. Among them, training samples containing multiple training images and their corresponding real labels, N i Human-like interaction and containing N q A set of query vectors Q with randomly initialized query vectors p Training the model, which includes the convolutional neural network, the Transformer encoder, the human-object pair decoder, the interactive action decoder, and the feedforward neural network, includes the following steps: Step T1: Randomly select training image I1 and training image I2 from the training samples, input training image I1 and training image I2 into the convolutional neural network for feature extraction, and then encode them through the Transformer encoder to obtain the global feature vector and position encoding vector of the training image respectively. Step T2: For each training image, combine the global feature vector, the location encoding vector, and the query vector Q. p Input the human-object pair decoder to obtain N q The feature vectors of the person-object pair are used as the feature vector set of the person-object pair; Step T3: For each training image, input the global feature vector, the location encoding vector, and the human-object pair feature vector group into the interactive action decoder to obtain the N. q The interaction action feature vectors of the person-object pair are used as the interaction action feature vector group; Step T4: For each training image, input the human-object pair feature vector group and the interaction action feature vector group into the feedforward neural network to obtain human-object pair prediction and interaction action prediction. Directly combine the human-object pair prediction and the interaction action prediction to obtain human-object interaction prediction. Step T5: For each training image, match the human-object interaction prediction with the real label corresponding to the training image according to the Hungarian algorithm to obtain the most accurate human-object interaction prediction. Step T6: Based on the most accurate human-object interaction prediction of the two training images, obtain the corresponding most accurate human-object pair feature vector and the most accurate interaction action feature vector. Perform cross-recombination or internal recombination on the most accurate human-object pair feature vector and the most accurate interaction action feature vector to obtain a recombined human-object interaction feature vector group. Input the recombined human-object interaction prediction into the interaction action classification feedforward network to obtain the recombined human-object interaction prediction. Recombinate the real labels corresponding to the two training images to obtain the recombined real labels. Step T7: Calculate the loss function based on the most accurate human-object interaction prediction and the real label of the two training images, as well as the reconstructed human-object interaction prediction and the reconstructed real label; and optimize the parameters of the interaction action decoder based on the loss function calculation result. Step T8: Repeat steps T1 to T7 until all training images in the training samples have optimized the parameters of the model, at which point the model training is complete. The most accurate human-object interaction prediction for the k-th training image For the most accurate prediction of the human-object interaction for the nth human-object pair in the kth training image, The most accurate human bounding box for the nth person-object pair in the kth training image. For the most accurate object bounding box of the nth person-object pair in the kth training image, o kn Let a be the most accurate object category confidence vector for the nth person-object pair in the kth training image. kn This represents the most accurate action category confidence vector for the nth person-object pair in the kth training image. The true number of human-object interaction instances in the k-th training image.

2. The human-object interaction action recognition method based on recombined sample learning according to claim 1, characterized in that: in, Step S1 includes the following sub-steps: Step S1-1: Input the image to be recognized into the convolutional neural network for feature extraction to obtain a visual feature map X. v ; Step S1-2, based on the visual feature map X v The position encoding vector E is obtained; Steps S1-3, the visual feature map X v The position encoding vector E is input into the Transformer encoder for encoding to obtain the global feature vector X. s .

3. The human-object interaction action recognition method based on recombined sample learning according to claim 1, characterized in that: in, The feedforward neural network includes a human-object classification feedforward network and an interactive action classification feedforward network. Step S4 includes the following sub-steps: Step S4-1, the interactive action feature vector group R i Input the human-object classification feedforward network to obtain the N q The person-object pair prediction; Step S4-2, convert the person-object pair feature vector group R p and the interactive action feature vector group R i The concatenation is then input into the interactive action classification feedforward network to obtain the N. q Predicting the interaction actions of the person-object pair.

4. The human-object interaction action recognition method based on recombined sample learning according to claim 1, characterized in that: in, Step S5 includes the following sub-steps: Step S5-1, for each person-object pair, according to the N i Human-object interaction involves multiplying the action category confidence vector and the object category confidence vector to obtain N. i Product of confidence levels; Step S5-2: For each person-object pair, select the product of the largest confidence scores as the maximum confidence score of the person-object pair, and use the object category and the action category corresponding to the maximum confidence score as the predicted object category and the predicted action category of the person-object pair, respectively. Step S5-3, for the N q The person-object pairs are sorted from highest to lowest maximum confidence to obtain the N. q The predicted results of person-object interactions.

5. The human-object interaction action recognition method based on recombined sample learning according to claim 1, characterized in that: in, When performing the cross-recombination based on the most accurate human-object interaction predictions of training image I1 and training image I2, step T6 includes the following sub-steps: Step T6-1: Based on the most accurate human-object interaction prediction of the training image I1 Obtain the most accurate human-object pair feature vector set of the training image I1 and the most accurate interactive action feature vector set Step T6-2: Based on the most accurate human-object interaction prediction of the training image I2 Obtain the most accurate person-object pair feature vector set of the training image I2 and the most accurate interactive action feature vector set Step T6-3, group the most accurate person-object pair feature vectors. of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one to obtain the recombined character interaction feature group. Step T6-4, group the most accurate person-object pair feature vectors. of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one to obtain the recombined character interaction feature group. Step T6-5, reassemble the character interaction feature group and the recombined character interaction feature group The interaction action predictions are obtained by inputting each action into the interaction action classification feedforward network. and interactive action prediction Step T6-6: Predict Y from the person-object pairs in the training image I1. p1 and the predicted interactive actions Combining these elements yields a recombinant human-object interaction prediction. Predict the person-object pair Y from the training image I2 p2 and the predicted interactive actions Combining these elements yields a recombinant human-object interaction prediction. The recombinant human-object interaction prediction and the recombinant human-object interaction prediction For the recombined human-object interaction prediction; Step T6-7, based on the true label corresponding to the training image I1 Get real-life labels for people and things and interactive action real tags Based on the true label corresponding to the training image I2 Get real-life labels for people and things and interactive action real tags Step T6-8, according to the N i Human-like interaction, where the human-object interaction is linked to real-world tags. and the real tags of the interactive actions By pairing and combining them one by one, the corresponding recombinant human-object interaction predictions can be obtained. Recombination True Label The real labels of the people and things and the real tags of the interactive actions By pairing and combining them one by one, the corresponding recombinant human-object interaction predictions can be obtained. Recombination True Label The recombinant real label and the recombined real label The recombinant real label.

6. The human-object interaction action recognition method based on recombined sample learning according to claim 1, characterized in that: in, When performing the internal reconstruction based on the most accurate human-object interaction predictions of the training images I1 and I2, step T6 includes the following sub-steps: Step T6-1: Based on the most accurate human-object interaction prediction of the training image I1 Obtain the most accurate human-object pair feature vector set of the training image I1 and the most accurate interactive action feature vector set Step T6-2: Based on the most accurate human-object interaction prediction of the training image I2 Obtain the most accurate person-object pair feature vector set of the training image I2 and the most accurate interactive action feature vector set Step T6-3, group the most accurate person-object pair feature vectors. of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one, and then the original combination of the person-object pair feature vector and the interaction action feature vector is removed to obtain the recombined person interaction feature group. Step T6-4, group the most accurate person-object pair feature vectors. of Each person-object pair feature vector is compared with the most accurate interaction action feature vector group. of The interaction action feature vectors are concatenated one by one, and then the original combination of the person-object pair feature vector and the interaction action feature vector is removed to obtain the recombined person interaction feature group. Step T6-5, reassemble the character interaction feature group and the recombined character interaction feature group The interaction action predictions are obtained by inputting each action into the interaction action classification feedforward network. and interactive action prediction Step T6-6: Predict Y from the person-object pairs in the training image I1. p1 and the predicted interactive actions Combining these elements yields a recombinant human-object interaction prediction. Predict the person-object pair Y from the training image I2 p2 and the predicted interactive actions Combining these elements yields a recombinant human-object interaction prediction. The recombinant human-object interaction prediction and the recombinant human-object interaction prediction For the recombined human-object interaction prediction; Step T6-7, based on the true label corresponding to the training image I1 Get real-life labels for people and things and interactive action real tags Based on the true label corresponding to the training image I2 Get real-life labels for people and things and interactive action real tags Step T6-8, according to the N i Human-like interaction, where the human-object interaction is linked to real-world tags. and the real tags of the interactive actions Pair them one by one, then remove the true labels corresponding to the training image I1. Obtain the corresponding recombinant human-object interaction prediction Recombination True Label The real labels of the people and things and the real tags of the interactive actions Pair them one by one, then remove the true labels corresponding to the training image I2. Obtain the corresponding recombinant human-object interaction prediction Recombination True Label The recombinant real label and the recombined real label The recombinant real label.

7. The human-object interaction action recognition method based on recombined sample learning according to claim 5 or 6, characterized in that: in, The combination of an object category in the real-world label of the person-object pair and an action category in the feature vector of an interactive action in the real-world label of the interactive action exceeds N. i During human-object interaction, the element value corresponding to the object category in the interaction action feature vector is set to 0 in the reconstructed real-world label. The combination of an object category in the real-world label and all action categories in the feature vector of an interactive action in the real-world label exceeds N. i During human-object interaction, the combination of the object category and the interaction action feature vector is removed from the reconstructed real-world labels. The combination of one object category in the real-world label of the person-object pair and all action categories in the feature vectors of all interactive actions in the real-world label of the interactive action exceeds N. i During human-object interaction, the object category in the reconstructed real label corresponds to an interaction action feature vector with all zeros.

8. The human-object interaction action recognition method based on recombined sample learning according to claim 1, characterized in that: in, In step T7, the formula for calculating the loss function is as follows: L=λ b ·L b +λ u ·L u +λ o ·L o +λ a ·L a , In the formula L b L u L o and L a These are the bounding box regression loss function, bounding box intersection-over-union (IoU) loss function, object class loss function, and action class loss function, respectively, λ. b , λ u , λ o and λ a Let L be the weight hyperparameters of the bounding box regression loss function, the bounding box intersection-over-union loss function, the object category loss function, and the action category loss function, respectively, and let L be the loss function. The formula for calculating the loss function result is as follows: L batch =ρ·L orig +(1-ρ)·L compo , In the formula L batch L is the result of the loss function calculation. orig The result of calculating the loss function L is given by the most accurate human-object interaction prediction and the true label. compo The loss function L is calculated for the recombined human-object interaction prediction and the recombined real label, where ρ is the weight hyperparameter that balances the two.

Images