A two-stage human-object interaction detection method based on an encoding-decoding architecture

Abstract

The present application belongs to the technical field of computer vision, and particularly relates to a two-stage human-object interaction detection method based on an encoding-decoding architecture. The present application realizes extraction of global context-level memory encoding and efficient fusion of instance-level memory encoding under a shared backbone network, discards a traditional fixed-length query sequence combined with a Hungarian matching training mode, and adopts a variable-length query sequence for one-to-one training prediction; fully mines the multi-layer decoder prediction results of DETR to improve target detection performance, and designs a matching pair-level attention mechanism module capable of adaptively learning the hidden relationship between humans and objects; the proposed method is tested on two datasets of V-COCO and HICO-DET, and the results exceed the best existing level.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a two-stage human-object interaction detection method based on an encoding and decoding architecture. Background Technology

[0002] In recent years, artificial intelligence (AI) has experienced rapid development. Researchers and industry investors are no longer limited to traditional visual perception tasks that identify objects by observing the external natural environment, such as object detection and semantic segmentation. Instead, they are increasingly interested in image semantic understanding tasks that can interpret the contextual relationships between objects and decipher the potential information in images or scenes. This includes human-object interaction detection tasks. Human-object interaction detection aims to correctly identify and locate instances of people and objects, and then uncover the interaction relationships between these pairs, simplifying relatively complex human activities into predictions of the <person, object, action> triple. As a fundamental task in image semantic understanding, it plays a significant role in advancing AI research from the observational perception level to the level of reasoning and understanding, while also promoting the development of many areas in computer vision, including action understanding and visual question answering. The following section will detail the relevant background technologies in this field.

[0003] (1) Introduction to Human Interaction Detection Dataset

[0004] There are many publicly available datasets for human-object interaction detection tasks, with V-COCO and HICO-DET datasets, based on single RGB images, being commonly used. V-COCO is a common evaluation dataset for human-object interaction detection, containing 2533 training images, 2867 validation images, and 4946 test images. Object classes are taken from images already present in the MS-COCO dataset; simplified vocabulary is used to specify 26 common actions for different object classes. mAP is used to calculate the average AP of actions. In scenario one, the requirement is to accurately predict the absence of objects in the image; in scenario two, only the accuracy of the interaction action needs to be guaranteed. The HICO-DET dataset also uses 80 object categories and common verbs from the MS-COCO dataset to create interaction categories for each object, containing 37633 training images, 9546 test images, 127 actions, and 600 human-object interaction triples. Similarly, unlabeled images exist; mAP calculates the average AP of 600 triples; in the full mode, predictions on all images are considered, while in the known object mode, only the mAP of triples corresponding to accurately predicted objects in labeled images is counted.

[0005] (2) Transformer-based human-object interaction detection method

[0006] In recent years, due to the widespread application of deep learning attention technology in the field of computer vision, especially the rapid development of Transformer-based technology, human-object interaction detection methods have made great progress. The emergence of DETR has inspired many researchers. Researchers at HOI-Trans and QPIC have found that by adding an interaction action classification head and a human-object localization box head to the original two prediction heads of DETR, and relying on the Hungarian matching algorithm, it can meet the requirements of this field.

[0007] However, single-stage methods face the challenge of multi-task learning, and balancing object detection and action recognition tasks is a challenge. AS-NET proposes an instance-aware attention module that can fuse abstract features of instance branches and interaction branches and perform matching by predicting keypoint offsets. HOIR decouples the decoder and trains it in parallel, matching through HO pointers, while CDN decouples and cascades the original decoders. The instance decoder learns information about objects and people and passes it to the interaction action classification decoder through query embedding. DT goes a step further, decoupling the encoder and decoder into three parts, using a base decoder to generate unified features and an attention fusion module to assist instance and interaction decoder inference to improve results. CPC improves the Transformer's HOI detection by improving the training strategy and using enhanced decoding paths. That is, while predicting HOI, it uses human-object features to predict interactions, human-interaction features to predict objects, and object-interaction features to predict people for reinforcement learning in three tasks.

[0008] As the application of Transformer deepens, some researchers are attempting to integrate textual features into existing visual features to improve performance. OCN proposed a verb semantic module that can semantically enhance pre-extracted verb features and perform inference through multimodal calibration and fusion with visual features. CATN, through exploration, discovered that using the prior textual embedding information of object categories provided by external object detectors as the initialization of object query sequences helps improve results. GEN-VLKT utilizes the large-scale pre-trained multimodal model CLIP to extract the textual embeddings of labels to initialize the classifier and employs imitation loss to minimize the visual feature distance between the interactive decoder and CLIP.

[0009] The UPT scheme was the first to introduce the Transformer into two-stage human-object detection, proposing an odd-pair converter that can fuse the first-stage human and object features with the learned position codes, and obtain pairwise prediction results through cooperative and competitive encoders.

[0010] (3) Current Challenges in Human-Object Interaction Detection

[0011] First, training a one-stage HOI detection model is essentially a multi-task optimization problem. The inconsistent optimization directions of each sub-task can lead to convergence to a suboptimal solution. For HOI detection, at least four sub-tasks need to be optimized: bounding boxes for people, bounding boxes for objects, object categories, and action / interaction categories. This makes convergence more difficult than for object detection. Therefore, current one-stage solutions generally rely on pre-trained object detection model weights as initial weights and have slow convergence speeds, limiting the transfer, improvement, and generalization of the methods.

[0012] Secondly, existing two-stage methods largely lack the extraction of global contextual features to aid inference, leading to somewhat arbitrary reasoning. Two-stage methods fix the weights of the pre-trained object detector network and use its provided features and predictions, transforming the problem into how to extract human-object interaction information from existing features, avoiding the difficulties of multi-object optimization. However, there are assumptions: the appearance localization features provided by the object detector can indeed distinguish the similarities and differences in actions between different human-object interactions, and external environmental features have a relatively small impact on action recognition and can be discarded. However, interaction actions can be divided into direct contact actions and indirect contact actions. Some direct actions can be inferred from the surrounding environment, while indirect contact actions are usually not within the bounding boxes of people and objects, requiring additional inference features. Many existing methods introduce joint regions, text features, and skeletal points to enhance action recognition results, but neglect the benefits of extracting features directly from the image itself. How to process the features extracted in the first stage and design structures to further extract global contextual features from the image is also the subject of this invention. Summary of the Invention

[0013] This invention addresses the shortcomings of existing two-stage human-object interaction detection methods, such as insufficient extension of global context to aid reasoning and lack of a mechanism for attention feature fusion at the human-object matching level. It proposes a two-stage human-object interaction detection method based on an encoding-decoding architecture, aiming to improve results on public datasets.

[0014] The technical solution of the present invention:

[0015] A two-stage human-object interaction detection method based on an encoding-decoding architecture includes the following steps:

[0016] Step 100: Prepare the publicly available human-object interaction detection datasets from V-COCO and HICO-DET, respectively;

[0017] Step 200: Preprocess the dataset.

[0018] Furthermore, step 200 involves preprocessing the dataset, including the following steps:

[0019] Step 201: During the training phase, the image is randomly horizontally flipped;

[0020] Step 202: Adjust the image size and crop it randomly, or adjust the image size directly;

[0021] Step 203: Regularize the RGB channels of the images in the dataset, given the mean and variance;

[0022] Step 204: Adjust the image size and perform regularization during the inference phase;

[0023] Step 300: Construct the DETR object detector and train the object detection model on the dataset formed in step 100;

[0024] Specifically, the DETR object detector consists of three main parts: (1) a ResNet backbone network, which is used to extract a compact feature representation from the image to prevent memory explosion caused by subsequent attention operations; (2) a multi-layer stacked Transformer encoder-decoder that uses multi-head self-attention and feedforward networks. The Transformer encoder processes the features extracted by the backbone network with two-dimensional sinusoidal static position coding to form the Transformer encoder memory coding. The Transformer decoder uses a cross-attention mechanism to extract the useful part of the Transformer encoder memory coding and outputs the object query features; (3) the query object head (feedforward network) receives the object query features output by the Transformer decoder and outputs the prediction results.

[0025] It includes the following steps:

[0026] Step 301: Read the pre-trained parameter file and load the pre-trained model weights on the MS-COCO large-scale image object detection dataset;

[0027] Step 302: Read the images, object labels, and object bounding boxes from the dataset generated in step 200, and divide the dataset into training and validation sets.

[0028] Step 303: Input the training data into the ResNet backbone network in batches to obtain the corresponding feature maps, mask matrices and position codes;

[0029] Step 304: The feature map, mask matrix, and position encoding are processed by the Transformer encoder to generate corresponding encoder memory encodings for use by the decoder.

[0030] Step 305: Input the mask matrix and position encoding obtained in step 303, the Transformer encoder features obtained in step 304, and the query embedding and query sequence input into the Transformer decoder. Obtain the object query features through the cross attention mechanism, and then obtain the corresponding query prediction results through the query object head (feedforward network).

[0031] Step 306: During the training phase, the Hungarian bipartite graph matching method is used to match the real labels one-to-one; during the inference phase, the query results are obtained end-to-end without any post-processing.

[0032] Step 400: Using the dataset from Step 100, perform data preprocessing according to the requirements of the human-object interaction detection task;

[0033] Furthermore, in step 400, the dataset is preprocessed, with the main operations being the same as in step 200, but the brightness, contrast, and saturation of the images are changed during the training phase.

[0034] Step 500: Construct a two-stage human-object interaction detector based on an encoding / decoding architecture.

[0035] Specifically, a two-stage human-object interaction detection task can be regarded as a downstream task of the object detection task. Therefore, based on the DETR object detector in the first stage (i.e., step 300), three downstream modules are set up to construct the second-stage two-stage human-object interaction detector based on the encoder-decoder architecture. The three downstream modules are the global context encoder module, the matching pair level attention fusion module, and the matching pair interaction decoder.

[0036] The main function of the global context encoder module is to extract global context memory encodings that can be used for interaction detection from image features again, and to fuse them with local instance memory encodings to obtain fused memory encodings. The main function of the matching pair level attention fusion module is to fuse the features of different people and objects to form matching pair level features for subsequent inference. The matching pair interaction decoder fuses the fused memory encodings of the global context encoder module and the matching pair level features of the matching pair level attention fusion module through a cross-attention mechanism, and outputs the prediction results through the interaction action detection head (i.e., the feedforward layer).

[0037] Step 600 involves the structural design of a two-stage human-object interaction detector based on an encoder-decoder architecture. It utilizes the object detection results and features generated by the DETR object detector in step 300 to train downstream human-object interaction prediction. This includes the following steps:

[0038] Step 601: The query predictions and query features obtained by the DETR object detector multilayer decoder in the first stage are filtered for duplicate predictions and a certain number of human and object instances are selected after non-maximum suppression operation.

[0039] Step 602: The matching pair level attention fusion module in the two-stage human-object interaction detector based on the encoder-decoder architecture inputs the center position of each pair of human-object localization boxes, the height and width of the localization boxes, as well as the relative distance and overlap, into the matching pair position encoder to obtain the position code, and inputs it together with the filtered instance features into the matching pair level attention fusion module to obtain the matching pair level features.

[0040] Step 603: Based on the global context encoder module in the two-stage human-object interaction detector with the encoding and decoding architecture, the features output by the ResNet backbone network of the DETR target detector are encoded into global context memory codes by the global context encoder, and then fused with the local instance memory codes output by the Transformer encoder through cross-residual technology to obtain fused memory codes.

[0041] Specifically, the cross-residual technique refers to adding the features before the input to the Transformer encoder through a linear layer and the global context memory encoding output by the global context encoder, adding the features before the input to the global context encoder through a linear layer and the local instance memory encoding output by the Transformer encoder, and finally obtaining the fused memory encoding through a connection operation;

[0042] Step 604: Based on the matching pair interaction decoder module in the two-stage human-object interaction detector with the encoding and decoding architecture, the memory encoding and matching pair level features are fused and input into the matching pair interaction decoder. The interaction action detection head (i.e., the feedforward layer) outputs a multi-label action prediction result, which is multiplied by the previously predicted instance confidence as the interaction confidence.

[0043] The beneficial effects of this invention are:

[0044] This invention proposes a novel two-stage human-object interaction detection framework based on encoder-decoder, which achieves efficient fusion of global context-level memory encoding extraction and instance-level memory encoding under a shared backbone network. It abandons the traditional training mode of fixed-length query sequence combined with Hungarian matching and adopts variable-length query sequence for one-to-one training and prediction.

[0045] We fully leverage the prediction results of DETR's multi-layer decoder to improve object detection performance, and designed a matching pair-level attention mechanism module that can adaptively learn the hidden relationships between people and objects.

[0046] The proposed method was tested on the V-COCO and HICO-DET datasets, and the results surpassed the existing state-of-the-art. Attached Figure Description

[0047] Figure 1 This is a flowchart of the two-stage human-object interaction detection method based on an encoding-decoding architecture of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of the two-stage human-object interaction detector based on the encoding and decoding architecture provided by the present invention;

[0049] Figures 3(a) and 3(b) are schematic diagrams of the Transformer encoder and decoder in the DETR target detector, respectively.

[0050] Figure 4 This is a schematic diagram of the process structure of the matching pair-level attention module provided by the present invention. Detailed Implementation

[0051] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0052] like Figure 1 and Figure 2 As shown, the two-stage target detection method based on an encoding / decoding architecture provided in this embodiment includes the following steps:

[0053] Step 100: Prepare the publicly available human-object interaction detection datasets from V-COCO and HICO-DET, respectively;

[0054] Step 200: Preprocess the dataset;

[0055] Step 201: During the training phase, the image is randomly horizontally flipped;

[0056] Step 202: Adjust the image size and crop it randomly, or adjust the image size directly;

[0057] Step 203: Regularize the RGB channels of the images in the dataset, given the mean and variance;

[0058] Step 204: Adjust the image size and perform regularization during the inference phase;

[0059] Step 300: Construct the DETR object detector and train the object detection model on the dataset formed in step 100.

[0060] Specifically, the DETR object detector consists of three main parts: (1) a ResNet backbone network, which is used to extract a compact feature representation from the image to prevent memory explosion caused by subsequent attention operations; (2) a multi-layer stacked Transformer encoder-decoder that uses multi-head self-attention and feedforward networks. The Transformer encoder processes the features extracted by the backbone network with two-dimensional sinusoidal static position coding to form the Transformer encoder memory coding. The Transformer decoder uses a cross-attention mechanism to extract the useful part of the Transformer encoder memory coding and outputs the object query features; (3) the query object head (feedforward network) receives the object query features output by the Transformer decoder and outputs the prediction results.

[0061] Step 301: Read the pre-trained parameter file and load the pre-trained model weights on the MS-COCO large-scale image object detection dataset;

[0062] Step 302: Read the images, object labels, and object bounding boxes from the dataset generated in step 200, and divide the dataset into training and validation sets.

[0063] Step 303: The training data is input into the ResNet backbone network in batches. Since the image sizes differ in each batch, zero-padding is used to ensure that all images in a batch are of equal size. The ResNet backbone network reduces the dimensions of the images to 1 / 16 of their original dimensions, converting a batch of 3-channel RGB images into a 2048-channel feature map. Based on the zero-padding positions of the original images, the nearest neighbor interpolation operation is used to obtain the mask matrix of the feature map, and the corresponding positional encoding is generated using the two-dimensional sinusoidal static positional encoding method (see Formula 1), where pos represents the position of the word in the sentence, i represents the current dimension of the positional encoding, and d... model It has a fixed dimension and a size of 256;

[0064]

[0065] Step 304: The 2048-dimensional feature map is reduced to 256 dimensions using a 1*1 convolutional layer. The width and height of the feature matrix, along with the mask matrix and positional encoding, are input into the Transformer encoder. The Transformer encoder is a multi-layer stacked multi-head self-attention module. The query sequence, key sequence, and value sequence are all input feature matrices. After processing by linear layers with different initializations, the query sequence and key sequence are added to the positional encoding and then multiplied by a matrix. The result is divided by the square root of the number of channels, normalized by softmax, and then multiplied by the value sequence. Simultaneously, the value sequence is added as a residual and regularized. The final result is output through a feedforward layer, as shown in Equation 2 and Figure 3(b), where Q, K, and V are the query sequence, key sequence, and value sequence, respectively, and d kl It has a fixed dimension and a size of 256;

[0066]

[0067] Step 305: Define a fixed-length blank query sequence as the input query sequence, and use the learnable query embedding as the query sequence for the cross-attention mechanism. The Transformer encoded features obtained in step 304 are used as the memory-encoded input of the decoder, and are input together with the mask matrix and positional encoding obtained in step 303 as the key sequence and value sequence for the cross-attention mechanism. The cross-attention mechanism is the same as the self-attention mechanism mentioned in step 304, except that the query sequence is the initial blank sequence and the key sequence and value sequence are memory-encoded inputs. The cross-attention mechanism can transform memory-encoded inputs of different lengths into a fixed length. The output result is output through the self-attention layer and the feedforward layer. Finally, the corresponding query prediction result is obtained through the query object header (feedforward layer). The entire encoder layer is stacked in multiple layers, and each layer is optimized by auxiliary loss, as shown in Figure 3(a).

[0068] Step 306: During the training phase, a Hungarian bipartite graph matching method will be used to match the query results one-to-one with the true labels. During the inference phase, the query results will be obtained end-to-end without any post-processing.

[0069] Specifically, Hungarian bipartite graph matching defines a score function, similar to the definition of a loss function, as detailed in Formula 3, where... and b i These are the predicted and actual bounding box coordinates, respectively. To predict the category confidence, y is the actual label. For the prediction results, we select the query with the highest label score as the optimization target. Therefore, the number of predictions is relatively small during the inference stage, and the prediction results can be directly output without complex post-processing processes such as non-maximum suppression.

[0070]

[0071] Step 400: Using the dataset from Step 100, perform data preprocessing according to the requirements of the human-object interaction detection task.

[0072] Furthermore, in step 400, the dataset is preprocessed, with the main operations being the same as in step 200, but the brightness, contrast, and saturation of the images are changed during the training phase.

[0073] Step 500: Construct a two-stage human-object interaction detector based on an encoding / decoding architecture.

[0074] Specifically, a two-stage human-object interaction detection task can be regarded as a downstream task of the object detection task. Therefore, based on the DETR object detector in the first stage (i.e., step 300), three downstream modules are set up to construct the second-stage two-stage human-object interaction detector based on the encoder-decoder architecture. The three downstream modules are the global context encoder module, the matching pair level attention fusion module, and the matching pair interaction decoder.

[0075] The main function of the global context encoder module is to extract global context memory encodings that can be used for interaction detection from image features again, and to fuse them with local instance memory encodings to obtain fused memory encodings. The main function of the matching pair level attention fusion module is to fuse the features of different people and objects to form matching pair level features for subsequent inference. The matching pair interaction decoder fuses the fused memory encodings of the global context encoder module and the matching pair level features of the matching pair level attention fusion module through a cross-attention mechanism, and outputs the prediction results through the interaction action detection head (i.e., the feedforward layer).

[0076] Step 600 involves the structural design of a two-stage human-object interaction detector based on an encoder-decoder architecture. It utilizes the object detection results and features generated by the DETR object detector in step 300 to train downstream human-object interaction prediction. This includes the following steps:

[0077] Step 601: Set the first-stage DETR object detector to have 100 query sequences, so that 100*6 layers of query results can be obtained. After non-maximum suppression post-processing to filter duplicate predictions, select a certain number of human and object instances. Specifically, the number of instances should be greater than 0.2 and should not be less than 4 or more than 15. If this is violated, we will select instances with high confidence after sorting. The operation is performed separately for humans and objects.

[0078] Step 602: The matching pair level attention fusion module in the two-stage human-object interaction detector based on the encoder-decoder architecture inputs the center position of each pair of human-object bounding boxes, the height and width of the bounding boxes, as well as the relative distance and overlap, into the matching pair position encoder to obtain the position code, and inputs it together with the filtered instance features into the matching pair level attention fusion module to obtain the matching pair level features.

[0079] Specifically, the position encoder consists of multiple linear layers, increasing the dimension from 36 to 128, 256, 256, and 512, and then applying ReLU activation layers to obtain the position encoding. A matching pair-level attention fusion processor is referenced. Figure 4 Location code E spat The query sequence used as the matching pair-level attention fusion unit has a size of (N, N, 512), where N is the number of instances. Secondly, the key and value sequences represent the instance features (Feat_Pair) after the first stage of filtering. inst Its size is (N, 256). Unlike general cross-attention, we do not use matrix multiplication to calculate the attention at the matching pair level. Instead, we connect instance pairing and position encoding, pass them through a linear layer, and output the attention matrix W using a sigmoid function. pair Its size is (N, N, 1). Finally, it is compared with the copied value vector, Feat_Repeat. inst The features are multiplied and added together, and then enhanced by a feedforward FFN layer to obtain a feature matrix of size (N, 256), achieving early feature fusion. To improve performance, multi-layer stacking and multi-head attention techniques are employed. Finally, features from people and different objects are selected and concatenated to obtain a (M, 512) matching pair level feature Q. HOF , where M is the number of person-object matching pairs. See Formulas 4 to 6 for details;

[0080] W pair =Sigmoid(Concat(Feat_Pair) inst E spat (4)

[0081]

[0082]

[0083] Step 603: Based on the global context encoder module in the two-stage human-object interaction detector with the encoding and decoding architecture, the features output by the ResNet backbone network of the DETR target detector are encoded into global context memory codes by the global context encoder, and then fused with the local instance memory codes output by the Transformer encoder through cross-residual technology to obtain fused memory codes.

[0084] Specifically, the global context encoder of this invention can adopt, but is not limited to, a Transformer encoder architecture, employing a frozen ResNet backbone with shared DETR to provide features that are output as x through a linear layer. GCE After passing through the encoder, output M GCE The cross-residual technique will combine the features x from the first stage. DETR The mapping and M GCE Add them together, and x GCE With M DETR The mappings are added together, and the last two branches are concatenated to achieve the final output fused memory code M. CR The training is aided by integrating global and local information, see Formula 7 for details;

[0085] M CR =Concat(GCE(x) GCE +pos)+linear(x DETR ),linear(M DETR )+x GCE (7)

[0086] Step 604: Based on the matching pair interaction decoder module in the two-stage human-object interaction detector with the encoding and decoding architecture, the memory encoding and matching pair level features are fused and input into the matching pair interaction decoder. The interaction action detection head outputs a multi-label action prediction result, which is multiplied by the previously predicted instance confidence as the interaction confidence.

[0087] Specifically, the matching pair interactive decoder of this invention abandons the Hungarian bipartite graph matching algorithm for training and directly uses a variable-length query sequence, the length of which is the number of matching pairs M. The query sequence is the matching pair level feature Q. HOF The memory encoding uses the output M of the global context encoder module. CR The decoder is a Transformer decoder, and the output features are transformed into interactive prediction confidence V through a feedforward layer FFN, as detailed in Formula 8.

[0088] V = FFN(PID(Q) HOF M CR (8)

[0089] This invention first trains a DETR object detector during the training phase, then uses instance features provided in the first phase, processed by a global context encoder module, a matching level attention fusion module, and a matching interaction decoder, to train human interaction detection. Real-time inference is possible during the inference phase.

[0090] Experiments were conducted on the HICO-DET and V-COCO benchmark datasets.

[0091] For the HICO-DET dataset, in full mode, compared to the baseline method UPT, the method of this invention improves mAP from 31.66 / 32.31 to 33.89 / 35.15, a relative improvement of 7.0% / 8.8%, without introducing multimodal textual semantic information or employing a more robust backbone. Compared to the one-stage methods CDN and QPIC, which also employ Transformer-based methods, it achieves relative gains of 7.8% / 9.6% and 16.6% / 17.6%, respectively. In the known object mode, it achieves a relative gain of 6.6% / 8.86% compared to the baseline method.

[0092] On the V-COCO dataset, compared to the baseline method UPT, the mAP in scenario one improved from 59.0 / 60.7 to 67.9 / 70.0, an improvement of 15.1% / 15.4%, and the mAP in scenario two improved from 64.5 / 66.2 to 70.13 / 72.32, an improvement of 8.7% / 9.2%. Compared to the original best solution DT, there is also a relative improvement of 2.6%.

[0093] Testing has shown that the two-stage task interaction detection method based on an encoding / decoding architecture provided in this invention can achieve good results on open-source datasets and can be applied in real-world scenarios. Furthermore, the method addresses the current problem of inference lacking a global perspective, demonstrating originality.

Claims

1. A two-stage human-object interaction detection method based on an encoding-decoding architecture, characterized in that, Includes the following steps: Step 100: Prepare the publicly available human-object interaction detection datasets from V-COCO and HICO-DET, respectively; Step 200: Preprocess the dataset; Step 201: During the training phase, the image is randomly horizontally flipped; Step 202: Adjust the image size and crop it randomly, or adjust the image size directly; Step 203: Regularize the RGB channels of the images in the dataset, given the mean and variance; Step 204: Adjust the image size and perform regularization during the inference phase; Step 300: Construct the DETR object detector and train the object detection model on the dataset formed in step 100; Specifically, the DETR object detector consists of three parts: (1) a ResNet backbone network, which is used to extract a compact feature representation from the image to prevent memory explosion caused by subsequent attention operations; (2) a multi-layer stacked Transformer encoder-decoder with multi-head self-attention and feedforward networks. The Transformer encoder processes the features extracted by the backbone network with two-dimensional sinusoidal static position coding to form the Transformer encoder memory coding. The Transformer decoder uses a cross-attention mechanism to extract the useful part of the Transformer encoder memory coding and outputs the object query features; (3) the query object head receives the object query features output by the Transformer decoder and outputs the prediction results. It includes the following steps: Step 301: Read the pre-trained parameter file and load the pre-trained model weights on the MS-COCO large-scale image object detection dataset; Step 302: Read the images, object labels, and object bounding boxes from the dataset generated in step 200, and divide the dataset into training and validation sets. Step 303: Input the training data into the ResNet backbone network in batches to obtain the corresponding feature maps, mask matrices and position codes; Step 304: The feature map, mask matrix, and position encoding are processed by the Transformer encoder to generate corresponding encoder features for use by the decoder; Step 305: Input the mask matrix and position encoding obtained in step 303, the Transformer encoder features obtained in step 304, the query embedding and query sequence input into the Transformer decoder, obtain the object query features through the cross attention mechanism, and then obtain the corresponding query prediction results through the query object head, i.e., the feedforward network. Step 306: During the training phase, the Hungarian bipartite graph matching method is used to match the real labels one-to-one; during the inference phase, the query results are obtained end-to-end without any post-processing. Step 400: Using the dataset from Step 100, perform data preprocessing according to the requirements of the human-object interaction detection task; The main operations are the same as in step 200, but the brightness, contrast, and saturation of the image are changed during the training phase; Step 500: Construct a two-stage human-object interaction detector based on an encoding / decoding architecture; Specifically, a two-stage human-object interaction detection task can be regarded as a downstream task of the object detection task. Therefore, based on the DETR object detector in the first stage (step 300), three downstream modules are set up to construct the second stage two-stage human-object interaction detector based on the encoder-decoder architecture. The three downstream modules are the global context encoder module, the matching pair level attention fusion module, and the matching pair interaction decoder. Specifically, the global context encoder module extracts global context memory codes from image features that can be used for interactive detection, and can fuse them with local instance memory codes to obtain fused memory codes; The role of the matching pair level attention fusion module is to fuse the features of different people and objects to form matching pair level features for subsequent inference; while the matching pair interaction decoder fuses the fusion memory encoding of the global context encoder module and the matching pair level features of the matching pair level attention fusion module through a cross attention mechanism, and outputs the prediction result through the interaction action detection head. Step 600 involves the structural design of a two-stage human-object interaction detector based on an encoder-decoder architecture. It utilizes the target detection results and features generated by the DETR target detector in step 300 to perform downstream human-object interaction prediction training; this includes the following steps: Step 601: The query predictions and query features obtained by the DETR object detector multilayer decoder in the first stage are filtered for duplicate predictions and a certain number of human and object instances are selected after non-maximum suppression operation. Step 602: The matching pair level attention fusion module in the two-stage human-object interaction detector based on the encoder-decoder architecture inputs the center position of each pair of human-object localization boxes, the height and width of the localization boxes, as well as the relative distance and overlap, into the matching pair position encoder to obtain the position code, and inputs it together with the filtered instance features into the matching pair level attention fusion module to obtain the matching pair level features. Step 603: In the global context encoder module of the two-stage human-object interaction detector based on the encoding and decoding architecture, the features output by the ResNet backbone network of the DETR target detector are encoded into global context memory codes by the global context encoder, and then fused with the local instance memory codes output by the Transformer encoder through cross-residual technology to obtain fused memory codes. Specifically, the cross-residual technique refers to adding the features before the input to the Transformer encoder through a linear layer and the global context memory encoding output by the global context encoder, adding the features before the input to the global context encoder through a linear layer and the local instance memory encoding output by the Transformer encoder, and finally obtaining the fused memory encoding through a connection operation; Step 604: Based on the matching pair interaction decoder module in the two-stage human-object interaction detector with the encoding and decoding architecture, the memory encoding and matching pair level features are fused and input into the matching pair interaction decoder. The interaction action detection head outputs a multi-label action prediction result, which is multiplied by the previously predicted instance confidence as the interaction confidence.

Images