A pre-training method for human-object interaction detection

Abstract

The present invention discloses a pre-training method for human-object interaction detection, comprising the following steps: obtaining datasets for target detection and action recognition, respectively, and performing enhancement processing on the datasets; constructing and initializing a pre-training network, presetting a loss function, and supervising the outputs of the target detection network and action recognition network of the pre-trained network based on the loss function; and using the trained pre-training network parameters as initialization weights for the human-object interaction detection network. Based on the concept of decoupling, the present invention decouples the pre-training process into two subtasks of human-object interaction detection, thereby obtaining a large amount of data with existing annotated information for pre-training. Both subtasks benefit from the clean annotated information, and after fine-tuning, the accuracy of human-object interaction detection can be significantly improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a pre-training method for detecting human-object interaction. Background Art

[0002] Human-object interaction detection can detect the location, object type, and interaction relationship of people and objects interacting in an image. Human-object relationship detection is crucial for applications such as autonomous driving, where it helps determine road conditions by monitoring the relationships between pedestrians and surrounding objects, enabling safe driving. In hospital monitoring systems, it can assess the patient's health and safety by determining whether they are in an emergency situation based on the relationships between them and surrounding objects.

[0003] Due to the Transformer's powerful ability to extract contextual features, current mainstream human-object interaction detection methods are typically based on the Detection Transformer (DETR) structure. Since DETR training requires a large amount of data, most existing methods use DETR weights pre-trained on object detection datasets to initialize model parameters. However, pre-trained DETR weights do not contain motion information, which greatly limits the ability of human-object interaction detection models to learn better motion features and more accurately predict interaction categories. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a pre-training method for human-object interaction detection. Based on the decoupling idea, the present invention decouples the pre-training process into two sub-tasks in human-object interaction detection, thereby obtaining a large amount of data with existing labeled information for pre-training. Both sub-tasks can benefit from clean labeled information, and after fine-tuning, the accuracy of human-object interaction detection can be significantly improved.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a pre-training method for detecting human-object interaction, comprising the following steps:

[0007] Obtain the datasets for target detection and action recognition respectively, and perform enhancement processing on the datasets;

[0008] Build a pre-trained network and initialize the pre-trained network, including:

[0009] Build and initialize a deep neural network based on CNN, and input the training images of the enhanced dataset into the deep neural network to obtain the feature map F;

[0010] Build and initialize the Transformer encoder, build the position encoding PE of the input image features, input the feature map F and the position encoding PE into the Transformer encoder to obtain the image features E;

[0011] Construct and initialize the Transformer detection decoder, input the updated image feature E and position encoding PE of the Transformer encoder into the Transformer detection decoder to obtain the output feature vector set D o ;

[0012] Set the threshold for filtering reliable human instances and output the feature vector set D o After threshold filtering, we get reliable person instance Q rp ;

[0013] Build and initialize the Transformer interaction decoder to convert the reliable person instance Q rp , picture features E, position encoding PE input Transformer interactive decoder to obtain the output feature vector set D a ;

[0014] Construct and initialize the target detection network and output the feature vector set D o The target detection network predicts the location and category of objects in the target detection dataset, and matches them with the labeled object instances in the target detection dataset based on the Hungarian algorithm.

[0015] Construct and initialize the action recognition network and output the feature vector set D a After the action recognition network is used, the categories of action interactions in the action recognition images to be tested in the action recognition dataset are predicted;

[0016] Preset loss function, and supervise the output of the object detection network and action recognition network of the pre-trained network based on the loss function;

[0017] Use the trained pre-trained network parameters as the initial weights of the human-object interaction detection network.

[0018] As a preferred technical solution, the feature map F and the position code PE are input into the Transformer encoder to obtain the image feature E, which is specifically expressed as:

[0019] E=f enc (F,PE)

[0020] Among them, f enc Represents the Transformer encoder module;

[0021] The Transformer encoder includes multiple cascaded encoder layers, each of which is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module. The query matrix Q of the self-attention module in the encoder is e , key matrix K e Sum matrix V e They are F+PE, F+PE and F respectively.

[0022] As a preferred technical solution, the updated image feature E and position code PE of the Transformer encoder are input into the Transformer detection decoder to obtain the output feature vector set D o , specifically expressed as:

[0023] D o =f det (Q,E,PE)

[0024] Among them, f det Represents the Transformer detection decoder module, Q represents a set of learnable vectors;

[0025] The Transformer detection decoder includes multiple cascaded detection decoder layers. Each detection decoder layer consists of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network, and a layer normalization processing module. The query matrix, key matrix, and value matrix of the mutual attention module in the detection decoder are Q, E+PE, and E, respectively, which are specifically expressed as:

[0026]

[0027] K d =E+PE

[0028] V d =E

[0029] Among them, N q is the query vector q in the query matrix of the mutual attention module i The number of .

[0030] As a preferred technical solution, the position encoding PE of the input image feature is constructed as follows:

[0031]

[0032]

[0033] Among them, pos represents a position in the two-dimensional image, d is a constant, j represents the dimension, for positions where the channel is odd, the cos function is used for position encoding; for positions where the channel is even, the sin function is used for position encoding. The final output position encoding PE is a three-dimensional position encoding matrix with the same dimension size as the feature map F.

[0034] As a preferred technical solution, the reliable person instance Q rp , picture features E, position encoding PE input Transformer interactive decoder to obtain the output feature vector set D a , specifically expressed as:

[0035] D a =f int (Q rp ,E,PE)

[0036] Among them, f int Represents the Transformer interactive decoder module;

[0037] The Transformer interactive decoder includes multiple cascaded interactive decoder layers, each of which is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module.

[0038] As a preferred technical solution, the output feature vector set D o After threshold filtering, we get reliable person instance Q rp , specifically expressed as:

[0039]

[0040]

[0041]

[0042] Among them, f person For the human classification network in the target detection network, N is the confidence score of the output feature classification as human. Q is the number of learnable vector sets, t is the threshold for filtering reliable human instances, and I is the index set that meets the threshold condition.

[0043] As a preferred technical solution, the target detection network includes two forward feedback networks, which respectively predict the position and category of the object. The two forward feedback networks are composed of three fully connected layers and one fully connected layer, which are specifically expressed as follows:

[0044]

[0045]

[0046] Among them, F o It consists of three fully connected layers, and the activation function between the fully connected layers is ReLU, F c is a fully connected layer;

[0047] Output feature vector set D o After the target detection network, the target detection network obtains the prediction result through matrix multiplication. The j-th prediction result triplet is expressed as in, is the normalized position of the object box, Indicates the category of the object, N obj is the number of object categories described in the target detection dataset.

[0048] As a preferred technical solution, the action recognition network includes a forward feedback network for predicting the category of action interaction. The forward feedback network is composed of a fully connected layer, which is specifically expressed as follows:

[0049]

[0050] Among them, F a Consists of 1 fully connected layer;

[0051] The action recognition network obtains the prediction result through matrix multiplication. The j-th prediction result triplet is expressed as in, represents the action category of the j-th prediction result, N act is the number of interaction categories described in the action recognition dataset.

[0052] As a preferred technical solution, the output of the target detection network and the action recognition network based on the loss function supervision pre-trained network specifically includes:

[0053] The loss function of the target detection network is specifically:

[0054]

[0055]

[0056]

[0057] in, Represents the set of non-empty set subscripts of the relationship pairs annotated in the image, Represents its quantity, Φ represents the set of empty set subscripts of object instances marked in the picture, Label the corresponding target detection image after the target detection network is matched by the Hungarian algorithm The prediction results, N q N is the number of potential object instances predicted by the target detection network in the target detection dataset. obj is the number of object categories described in the target detection dataset, L b represents L1 loss, L u represents the generalized IoU loss, L c Represents cross-entropy loss;

[0058] The loss function of the action recognition network is:

[0059]

[0060] Among them, l f is the element-by-element focal loss, is the prediction result after action recognition network fusion, a i is the annotation of the action recognition dataset, N a The number of predicted outcomes for the final action;

[0061] When focal loss is used to supervise the output of the action recognition network, the way to fuse the action prediction results is as follows:

[0062] When reliable character example Q rp The action prediction results are obtained through the action recognition network Among them, N rp and N act Represents the reliable person instance Q rp The number of and the number of action categories in the action recognition dataset;

[0063] The output prediction results are:

[0064]

[0065] in, max represents the maximum pooling operation.

[0066] As a preferred technical solution, when the input action recognition dataset is a video dataset, the fusion method of the output results is:

[0067] When it is determined that the action changes between multiple frames of the input video dataset are relatively smooth, each video segment in the video dataset is uniformly sampled into multiple frames to convert them into an image dataset, and processed according to the method of action recognition network recognition dataset;

[0068] When it is determined that the action changes between multiple frames of the input video data set are more drastic, each video segment is processed into multiple frames and input into the pre-trained network together to obtain the action prediction result The final output prediction result is:

[0069]

[0070] Among them, the prediction results The focal loss is calculated based on the annotation information provided by the action recognition video dataset.

[0071] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0072] (1) Based on the idea of ​​decoupling, the present invention decouples the pre-training process into two subtasks in human-object interaction detection and obtains corresponding prediction result triplets, thereby obtaining a large amount of data with existing labeled information for pre-training. Both subtasks can benefit from clean labeled information, and after fine-tuning, the accuracy of human-object interaction detection can be significantly improved.

[0073] (2) The present invention uses reliable human instance queries for action recognition, which solves the alignment problem between the pre-trained model framework and the downstream fine-tuning model framework, making the paradigm of the pre-training process and the paradigm of fine-tuning as consistent as possible, so that the pre-trained knowledge can be better transferred. In addition, using reliable human instance queries for action recognition also enables the model to pay more attention to human-centric action feature information, which is very important for the task of human-object interaction detection.

[0074] (3) The present invention adopts a strategy of fusion of action prediction results, and uses the maximum pooling operation on the output of the action classifier to obtain the fused prediction result for loss function calculation, which solves the problem that the action recognition dataset only has the action label of the whole picture during the pre-training process. The strategy of fusion of action prediction results can not only use positive samples for stable training, but also suppress the training noise that may be generated by difficult negative samples.

[0075] (4) The present invention utilizes a large number of action recognition data sets for pre-training and designs a reasonable pre-training model framework and loss function calculation strategy, thereby solving the problem that the pre-training methods used in existing human-object interaction detection only contain target detection information but lack action recognition-related knowledge, thereby greatly improving the detection accuracy of the human-object interaction detection model during fine-tuning.

[0076] (5) During the fine-tuning process, the present invention only needs to train the pre-trained network parameters once to perform fine-tuning on various human-object interaction detection methods; for mainstream human-object interaction detection methods, most of the parameters in the pre-training process can be directly loaded for fine-tuning; for a small number of methods with customized interaction detection modules, CNN convolutional neural networks and Transformer encoders can be loaded for fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of the process of the present invention's pre-training method for detecting human-object interaction;

[0078] Figure 2 Schematic diagram of the basic network structure of the Transformer encoder of the present invention;

[0079] Figure 3 Schematic diagram of the basic network structure of the Transformer detection decoder of the present invention;

[0080] Figure 4 Schematic diagram of query vector input for the Transformer detection decoder and interactive decoder of the present invention;

[0081] Figure 5 Schematic diagram of the basic network structure of the target detection network of the present invention;

[0082] Figure 6 Schematic diagram of the basic network structure of the action recognition network of the present invention. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0084] Example 1

[0085] like Figure 1 As shown, this embodiment provides a pre-training method for human-object interaction detection. This method decouples the pre-training process into two subtasks of human-object interaction detection and aligns the pre-training process with the human-object interaction detection process through reliable human examples. This greatly improves the accuracy of the fine-tuned human-object interaction detection model. The method specifically includes the following steps:

[0086] S1: Obtain the datasets for target detection and action recognition respectively, and perform enhancement processing on the datasets;

[0087] In this embodiment, the acquired object detection datasets include MS-COCO and Objects365, the action recognition datasets include image action recognition datasets and video action recognition datasets, the image action recognition datasets include HAKE and MPII, and the video action recognition datasets include Haa500 and Kinetics-700.

[0088] In this embodiment, the data set is enhanced, specifically including: randomly flipping the image horizontally, jittering the color, scaling the image, cropping the image, and finally normalizing the image;

[0089] In this embodiment, data enhancement is performed on the data of all data sets, specifically: for an input image, it is horizontally flipped with a probability of 50%; its brightness, contrast and saturation are jittered in the range of [0.6, 1.4]; then it is scaled with a probability of 50%, and the shortest side is selected with a medium probability in [480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800] during the scaling, while ensuring that the longest side does not exceed 1333; finally, the image is normalized, and the normalized mean and variance are [0.485, 0.456, 0.406] and [0.229, 0.224, 0.225], respectively.

[0090] S2: Build a pre-trained network and initialize the pre-trained network;

[0091] In this embodiment, constructing a pre-trained network and initializing the pre-trained network specifically includes:

[0092] S21: Build and initialize a deep neural network based on CNN;

[0093] In this embodiment, the method for constructing a deep neural network is:

[0094] The CNN-based deep neural network constructed uses a residual network ResNet-50 or ResNet-101, followed by a 1x1 convolution to reduce the number of channels. The initialization method uses the parameters of the ResNet-50 object classification model trained on ImageNet as the initialization parameters;

[0095] For a data-enhanced training image, a feature map F is first obtained through a CNN-based deep neural network;

[0096] S22: Build and initialize the Transformer encoder, obtain the feature map F and input it into the Transformer encoder;

[0097] In this embodiment, the Transformer encoder is constructed as follows:

[0098] like Figure 2 As shown, an encoder is formed by l cascaded encoder layers. In this embodiment, l is preferably 6. Each encoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module. The query matrix Q of the self-attention module in the encoder is e , key matrix K e Sum matrix V e They are F+PE, F+PE and F respectively. The output of the Transformer encoder is E, which is specifically expressed as:

[0099] E=f enc (F,PE)

[0100] Among them, E is the image feature extracted by CNN and further updated by Transformer encoder, f enc represents the Transformer encoder module, PE is the position encoding of the input image features, and F is the feature map extracted after the input image passes through CNN;

[0101] S23: Build and initialize the Transformer detection decoder, and input the updated image features E and position encoding PE into the Transformer detection decoder;

[0102] In this embodiment, the Transformer detection decoder is constructed as follows:

[0103] like Figure 3 As shown, a detection decoder is formed by l cascaded detection decoder layers. In this embodiment, l is preferably 6, which is consistent with the number of layers of the encoder. Each detection decoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module. The query matrix, key matrix and value matrix of the mutual attention module in the detection decoder are Q, E+PE and E respectively, which are specifically expressed as follows:

[0104]

[0105] K d =E+PE

[0106] V d =E

[0107] Among them, N qis the query vector q in the query matrix of the mutual attention module i The number of

[0108] The output feature vector set of the Transformer detection decoder is D o , specifically expressed as:

[0109] D o =f det (Q,E,PE)

[0110] Among them, F represents the output features of the input image after the deep neural network based on CNN, PE represents the position encoding of the input image, Q represents a set of learnable vectors, and f det Represents the Transformer detection decoder module;

[0111] In this embodiment, it is necessary to perform three-dimensional position encoding on each pixel of the three-dimensional feature map E. The position encoding method of the Transformer is:

[0112]

[0113]

[0114] Among them, pos represents a position of a two-dimensional image, d is a constant. In this embodiment, d is preferably 128, j represents the dimension, and for positions where the channel is an odd number, the cos function is used for position encoding; for positions where the channel is an even number, the sin function is used for position encoding. The final output PE is a three-dimensional position encoding matrix with the same dimension size as F.

[0115] In this embodiment, the initialization method of the Transformer detection decoder is random initialization;

[0116] S24: Build and initialize the Transformer interaction decoder;

[0117] In this embodiment, the Transformer interaction decoder is constructed as follows: like the Transformer detection decoder, it has l cascaded interaction decoder layers, and each interaction decoder layer is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module.

[0118] The calculation process of the Transformer interactive decoder is expressed as:

[0119] D a =f int (Q rp,E,PE)

[0120] Among them, f int Denotes the Transformer interactive decoder module, D a is the set of output feature vectors of the Transformer interaction decoder, and the Transformer interaction decoder is initialized randomly.

[0121] like Figure 4 As shown, in the Transformer interactive decoder, the query matrix of the mutual attention module is a set of reliable human instance visual features Q rp ;

[0122] In this embodiment, the query matrix Q of the mutual attention module in the Transformer interactive decoder is rp The construction method is:

[0123]

[0124]

[0125]

[0126] Where, f person For the human classification network in the target detection network, N is the confidence score of the output feature classification as human. Q is the number of learnable vector sets Q, t is the threshold for filtering reliable human instances, I is the index set that meets the threshold condition, and N in this embodiment Q and t are preferably 100 and 0.9, respectively.

[0127] S25: Figure 5 As shown in the figure, a target detection network is constructed, and the position and category of objects in the target detection dataset are predicted based on the output of the Transformer detection decoder, and the target detection network is initialized.

[0128] In this embodiment, the object detection network includes two feedforward networks, which predict the position and category of the object respectively. The two feedforward networks are composed of three fully connected layers (with intermediate ReLU activation functions) and one fully connected layer, which are specifically expressed as follows:

[0129]

[0130]

[0131] Among them, F o It consists of three fully connected layers, and the activation function between the fully connected layers is ReLU, Fc is a fully connected layer, F o and F c The initialization method is random initialization.

[0132] In this embodiment, the position and category of the object in the target detection picture are predicted based on the output of the Transformer detection decoder, and the j-th prediction result obtained specifically includes:

[0133] When the input is target detection data, the output feature vector set D of the detection decoder o After the target detection network, the target detection network finally obtains the prediction result through matrix multiplication. The j-th prediction result triplet includes in is the normalized position of the object box, Indicates the category of the object, N obj is the number of object categories described in the target detection dataset.

[0134] When the input is action recognition data, D o Filter by a threshold to get reliable person instance Q rp , and the Transformer encoder feature E and position encoding PE are input into the Transformer interaction decoder.

[0135] In this embodiment, the input images can be a batch, for example, 16 images, there will be 8 target detection data sets and 8 action recognition data sets, and the first 8 images can be fixed as target detection data sets and the last 8 images as action recognition data sets.

[0136] S26: Figure 6 As shown in the figure, an action recognition network is constructed, and the interaction category in the test image of the action recognition dataset is predicted based on the output of the Transformer interaction decoder, and the action recognition network is initialized;

[0137] In this embodiment, the category of the interaction in the action recognition picture to be tested is predicted based on the output of the Transformer interaction decoder, and the j-th prediction result obtained specifically includes:

[0138] When the input is action recognition data, D a After the action recognition network, the action recognition network finally obtains the prediction result through matrix multiplication. The j-th prediction result triplet includes in, represents the action category of the j-th prediction result, N act The number of interaction categories described in the action recognition dataset;

[0139] In this embodiment, the action recognition network includes a feed-forward network for predicting the category of action interaction. The feed-forward network consists of a fully connected layer, which is specifically expressed as follows:

[0140]

[0141] Among them, F a It consists of 1 fully connected layer, F a The initialization method is random initialization;

[0142] In this embodiment, after obtaining the prediction results of the target detection network, the prediction results (the location and category of the object) are matched with the labeled object instances in the target detection dataset using the Hungarian algorithm, where the loss matrix is ​​calculated as follows:

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] in, Represents the i-th annotated object instance in the image, which are the object box information and category information annotated by the real label, Φ represents the set of empty set subscripts of the object instances annotated in the image; GIOU is the abbreviation of generalizedIoU; N ogt is the number of instances annotated in the current sample of the target detection dataset, is the normalized position of the object box, Indicates the category of the object, st indicates the constraint condition. This embodiment adopts cross-entropy los, and the constraint condition is that the probability distribution sum is 1, that is, the prediction result must be scaled and normalized by the softmax function. After the above calculation, the subscript position of the predicted object instance result corresponding to each marked object is obtained.

[0149] S3: Use the preset loss function to supervise the output of the object detection network branch and the action recognition network branch of the pre-trained network during pre-training;

[0150] In this embodiment, the preset loss functions include: L1 loss, generalized IoU loss, cross-entropy loss, and focal loss;

[0151] Among them, L1 loss and generalized IoU loss are used to supervise the object box regression of the target detection branch; cross-entropy loss is used to supervise the object category classification of the target detection branch; focal loss is used to supervise the interaction category classification of the action recognition branch.

[0152] After obtaining the prediction output of the target detection network, multiple loss functions are used to supervise it. The specific target detection branch loss function is:

[0153]

[0154]

[0155]

[0156] in, Represents the set of non-empty set subscripts of the relationship pairs annotated in the image, represents its quantity; Φ represents the set of empty set subscripts of object instances marked in the image; Label the corresponding target detection image after the target detection network is matched by the Hungarian algorithm The prediction results of N q N is the number of potential object instances predicted by the target detection network in the target detection dataset. obj The number of object categories described in the target detection dataset;

[0157] After obtaining the prediction output of the action recognition network, the focal loss function is used to supervise it. The specific action recognition branch loss function is:

[0158]

[0159] l f It is the element-by-element focal loss; is the prediction result after the action recognition branch is fused; a i Annotation for action recognition dataset; N a is the number of final action prediction results, which is always 1 after fusion.

[0160] The final total loss function is:

[0161] L=λ b L b +λ u L u +λ c L c +λ a L a

[0162] L b 、L u 、L c 、L a are L1 loss, generalized IoU loss, cross-entropy loss and focal loss of the pre-trained network respectively; λ b ,λ u ,λ c ,λ a is the weight; since the target detection data and action recognition data are input into the network at the same time, all the above losses will be calculated in one update process of the model. b ,λ u ,λ c ,λ a Preferably, they are 2.5, 1, 1 and 1 respectively;

[0163] In this embodiment, when focal loss is used to supervise the output results of the action recognition branch, the method of fusing the action prediction results is as follows:

[0164] When Q rp A set of action prediction results are obtained through the action recognition branch Among them, N rp and N act Represents Q rp The number of and the number of action categories in the action recognition dataset;

[0165] The final prediction result is:

[0166]

[0167] in, max represents the maximum pooling operation. The focal loss can be calculated with the annotation information provided by the action recognition dataset;

[0168] In this embodiment, when the input action recognition dataset is a video dataset, the fusion method of the output results is:

[0169] When the action changes between multiple frames of the input video dataset are relatively smooth, each video segment in the video dataset is uniformly sampled into multiple frames to convert it into an image dataset, and processed according to the above-mentioned method for image action recognition dataset;

[0170] When the motion changes between multiple frames of the input video dataset are more drastic, each video segment is processed into 16 frames and input into the pre-trained network to obtain the motion prediction results. The final output prediction result is:

[0171]

[0172] Where, at this time The focal loss can be calculated with the annotation information provided by the action recognition video dataset.

[0173] S4: During fine-tuning, the pre-trained network parameters are used as the initial weights of the human-object interaction detection network;

[0174] During the fine-tuning process, it is only necessary to train the pre-trained network parameters once to perform fine-tuning on various human-object interaction detection methods; for the current mainstream human-object interaction detection methods, the parameters in the pre-training process of the present invention can be directly loaded for fine-tuning; for a small number of methods with customized interaction detection modules (that is, they may not include the Transformer detection decoder and Transformer interaction decoder with the same structure as this embodiment), the CNN convolutional neural network and Transformer encoder (these two modules usually have the same structure) can be loaded for fine-tuning.

[0175] To verify the effectiveness of the present invention, a fine-tuning experiment was conducted on the HICO-DET dataset, as shown in Table 1 below. The mean average precision (mAP) metric was used. The mAP was calculated by calculating the prediction accuracy for each action category included in the dataset across all test images. The average prediction accuracy of all actions is the mAP.

[0176] Table 1 Comparative data of fine-tuning of the present invention and other pre-training methods on HICO-DET

[0177] Methods full rare non-rare QPIC 29.07 21.85 31.23 CDN 32.06 27.75 33.35 GEN-VLKT 33.75 29.25 35.10 Ours(QPIC) 30.23 25.27 31.71 Ours(CDN) 34.30 31.16 35.23 Ours(GEN-VLKT) 34.67 30.11 36.03

[0178] As shown in Table 1, when the pre-training parameters obtained by the pre-training method proposed in the present invention are fine-tuned, the performance of the human-object interaction detection model is significantly improved, especially the performance of the rare category (rare category, the number of training samples is less than 10). Specifically, when QPIC applies the present invention, the performance of the rare category is improved by 3.42% of the average accuracy. At the same time, the performance of the full, rare and non-rare categories on CDN are improved by 2.24%, 3.41% and 1.88% of the average accuracy respectively. Moreover, when the present invention is used for GEN-VLKT, it can be observed that the performance on the full category continues to improve by 0.92% of the average accuracy. The above results show that the pre-training method proposed in the present invention can learn rich and action-related knowledge in the pre-training stage, thereby greatly improving the performance of the current human-object interaction detection model.

[0179] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A pre-training method for detecting human-object interaction, characterized in that: The steps include: Obtain the datasets for target detection and action recognition respectively, and perform enhancement processing on the datasets; Build a pre-trained network and initialize the pre-trained network, including: Build and initialize a deep neural network based on CNN, and input the training images of the enhanced dataset into the deep neural network to obtain the feature map F; Build and initialize the Transformer encoder to construct the positional encoding of the input image features , the feature map F and position encoding Input to Transformer encoder to get image features ; Build and initialize the Transformer detection decoder, and update the image features after the Transformer encoder and positional encoding Input Transformer detection decoder to get the output feature vector set ; Set the threshold for filtering reliable person instances and output a set of feature vectors Get reliable person instances through threshold filtering ; Build and initialize the Transformer interaction decoder to convert reliable character instances , picture features , position encoding Input Transformer interactive decoder to get the output feature vector set ; Build and initialize the target detection network and output a set of feature vectors The target detection network predicts the location and category of objects in the target detection dataset, and matches them with the labeled object instances in the target detection dataset based on the Hungarian algorithm. Build and initialize the action recognition network and output a set of feature vectors After the action recognition network is used, the categories of action interactions in the action recognition images to be tested in the action recognition dataset are predicted; Preset loss function, and supervise the output of the object detection network and action recognition network of the pre-trained network based on the loss function; Use the trained pre-trained network parameters as the initial weights of the human-object interaction detection network.

2. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The feature map F and position encoding Input to Transformer encoder to get image features , specifically expressed as: ； in, Represents the Transformer encoder module; The Transformer encoder includes multiple cascaded encoder layers, each of which is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module. The query matrix of the self-attention module in the encoder is , key matrix Sum Matrix They are , and .

3. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The image features after the Transformer encoder is updated and positional encoding Input Transformer detection decoder to get the output feature vector set , specifically expressed as: ； in, represents the Transformer detection decoder module, Represents a set of learnable vectors; The Transformer detection decoder includes multiple cascaded detection decoder layers, each of which is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network, and a layer normalization processing module. The query matrix, key matrix, and value matrix of the mutual attention module in the detection decoder are specifically expressed as: ； ； ； in, is the query vector in the query matrix of the mutual attention module The number of represents the query matrix, represents the bond matrix, Represents a matrix of values.

4. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The position encoding of the input image features is constructed , specifically expressed as: ； ； in, Represents a position in a two-dimensional image, is a constant, Represents the dimension. For odd-numbered channels, use Function for position encoding; for positions where the channel is even, use The function performs position encoding and finally outputs the position encoding is the three-dimensional position encoding matrix, dimension size and feature map consistent.

5. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The reliable character examples , picture features , position encoding Input Transformer interactive decoder to get the output feature vector set , specifically expressed as: ； in, Represents the Transformer interactive decoder module; The Transformer interactive decoder includes multiple cascaded interactive decoder layers, each of which is composed of a cascaded self-attention module, a residual connection network, a layer normalization processing module, a mutual attention module, a residual connection network, a layer normalization processing module, a forward feedback network, a residual connection network and a layer normalization processing module.

6. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The output feature vector set Get reliable person instances through threshold filtering , specifically expressed as: ； ； ； in, For the human classification network in the target detection network, To output the confidence score that the feature is classified as human, is the number of learnable vector sets, is the threshold for filtering reliable person instances, The index set that meets the threshold condition.

7. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The target detection network includes two feed-forward networks, which predict the position and category of the object respectively. Both feed-forward networks are composed of three fully connected layers and one fully connected layer, which are specifically expressed as follows: ； ； in, It consists of 3 fully connected layers, and the activation function between the fully connected layers is , is a fully connected layer; Output feature vector set After the target detection network, the target detection network obtains the prediction result through matrix multiplication. The prediction result triplet is expressed as , ,in, is the normalized position of the object box, Indicates the category of the object, is the number of object categories described in the target detection dataset.

8. The pre-training method for detecting human-object interaction according to claim 1, characterized in that: The action recognition network includes a feed-forward network for predicting the category of action interaction. A feed-forward network consists of a fully connected layer, which is specifically expressed as: ； in, Consists of 1 fully connected layer; The action recognition network obtains the prediction result through matrix multiplication. The prediction result triplet is expressed as , ,in, Indicates the The action category of the predicted results, is the number of interaction categories described in the action recognition dataset.

Images