A method for detecting character interaction relationship by concatenating decoding and double difficult sample mining

Abstract

The application discloses a kind of character interaction relationship detection methods of cascaded decoding and double difficult division sample mining, the method first uses picture context consistency splicing method, to the picture containing the character obtained is handled.Secondly based on the picture after processing, feature extraction is carried out through convolutional neural network, and then one-dimensional feature vector is obtained through flattening operation, and is recorded as input feature mapping.Finally, based on input feature mapping, human-object interaction relationship detection of cascaded decoding and double difficult division sample mining is carried out, and human-object interaction detection result is output.The application solves the problem of unbalanced training data samples, can accurately locate human-object pair, improve the effectiveness of interaction dependency relationship, and realize high-accuracy human-object interaction relationship detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of target detection, and is a person-object interaction relationship detection method based on cascade decoding and double-difficult sample mining. BACKGROUND

[0002] In real-world scenarios, human-object interaction is ubiquitous, such as reading a book, picking up an object, etc. Therefore, the research on human-object interaction relationship detection is of great significance. Especially for the tasks in the field of intelligent monitoring, human-computer interaction, etc., accurately predicting the human-object interaction relationship is a key step. In addition, accurately estimating the human-object interaction relationship can also provide information support for downstream visual understanding tasks, such as image retrieval, visual information-based question answering. Compared with target detection that independently identifies and locates objects, human-object interaction relationship detection focuses on capturing the complex association between people and objects, and human-object interaction detection has become a popular research direction.

[0003] However, the multi-element and complex interaction scenarios bring challenges to the performance improvement of human-object interaction relationship detection. The powerful feature learning and pattern recognition ability of deep learning is a powerful tool to complete the challenge of complex interaction detection in the real world. In order to further improve the performance and accuracy of human-object interaction relationship detection, human-object interaction relationship detection based on deep learning technology has attracted attention.

[0004] The traditional human-object interaction relationship detection method is a two-stage detection method. The traditional two-stage human-object interaction detection method is divided into two serial modules: a target detection module and an interaction relationship classification module. The target detection module first detects candidate human-object interaction pairs, and the interaction classification module predicts the interaction relationship class with a higher confidence score according to the human-object pairs output by the detection module. Compared with the single-stage detection method, the two-stage detection method does not realize efficient utilization of the information generated in the processing of the target detection module and the interaction relationship classification module, and does not realize the reduction of computational complexity. In addition, the number of candidate human-object pairs detected by the first-stage target detection module is usually greater than the number of actual human-object pairs involved in the interaction, and a large number of candidate human-object interaction pairs leads to increased computational cost. The inefficient data processing method produces redundant computational workload, and optimizing the data processing method helps to obtain a better human-object interaction detection model.

[0005] In recent years, the Transformer and its variants (DETR, ViT, etc.) have been successfully applied to a variety of visual tasks due to their strong predictive power and attention mechanisms. Currently, end-to-end human-object interaction detection methods based on DetectionTransformer (DETR) are gradually becoming mainstream, such as Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information (QPIC), Human-Object interaction TRansformer (HOTR), etc. Compared to two-stage methods based on convolutional neural networks, end-to-end single-stage methods use attention instead of convolution, effectively utilize global information for set prediction, and are driven by interactions. This method directly locates the interaction relationship or the human-object pair, effectively reducing the interference of weakly related human-object pairs. And the end-to-end method is more conducive to the detection system to capture the context information of human and object instances, such as the use of environmental information around the human-object pair.

[0006] Single-stage human-object interaction detection methods improve the overall efficiency of the model by addressing the high computational cost and low data processing efficiency of two-stage human-object interaction detection methods. Single-stage human-object interaction detection methods reduce the number of candidate human-object interaction pairs through an end-to-end approach, thereby reducing additional computational costs. Single-stage methods have obvious advantages in efficiency and other aspects compared to two-stage human-object interaction detection methods. However, the human-object pair detection and interaction relationship detection subtasks share the same decoder, which cannot fully capture effective dependencies, and the coupled human-object detection and interaction detection limit the performance of the model, making it difficult for the model to achieve excellent performance. Therefore, by improving the decoder in the above single-stage method, the ability of the model to capture dependencies is improved, thereby improving the detection ability of the model.

[0007] The purpose of hard sample mining is to find the most difficult-to-classify samples in the sample and train these samples as the focus to improve the performance of the model. Applying the hard sample mining method in human-object interaction detection methods helps to improve the robustness of the model, improve the performance of the model, and enhance the adaptability of the model to the uneven distribution of data in the feature space. SUMMARY

[0008] In view of the deficiencies of the prior art, the present application provides a single-stage human-object interaction relationship detection method in a complex scene. The present application uses an end-to-end human-object pair detection and interaction classification detection framework based on DetectionTransformer (DETR) to achieve efficient and accurate human-object interaction relationship detection. The human-object interaction relationship detection method based on cascaded decoding and double difficult sample mining proposed by the present application uses a cascaded decoding path to complete the target detection stage and the interaction relationship classification stage, thereby improving the accuracy of the single-stage human-object interaction relationship detection method. In addition, during the model training stage, the picture context consistency splicing method is used to alleviate the problem of uneven distribution of the average number of marked human-object pairs in the image and the long-tail problem of interaction class distribution under the premise of ensuring the authenticity of the sample, so that the distribution of human-object interaction pairs is closer to the cluster center. The present application replaces the spliced picture with the original sample picture, keeping the total amount of learning image samples unchanged. The present application improves the model's ability to capture dependency relationships by mining the query queue of difficult samples. The present application also uses an alternating joint sample mining strategy in the decoder part of DETR. Each training alternately selects a difficult sample mining method based on the true value box movement and a difficult sample mining method based on attention mapping masking to construct a difficult positive query queue and realize difficult positive query mining. Thus, the robustness of the single-stage human-object interaction relationship detection model is improved. The alternating joint learning strategy avoids using the difficult sample mining method based on the true value box movement and the difficult sample mining method based on attention mapping masking at the same time during training, which results in excessive computational load, so that the model has higher sample mining efficiency.

[0009] A human-object interaction relationship detection method based on cascaded decoding and double difficult sample mining includes the following steps:

[0010] Step 1: using a picture context consistency splicing method to process the obtained picture containing a person.

[0011] Set a probability threshold γ, randomly generate a random number between 0 and 1, and assign a probability value to each picture. If the probability value of the picture is greater than the probability threshold γ, the picture is preprocessed, otherwise it is not preprocessed.

[0012] The image in the data set is preprocessed using the picture context consistency splicing method. The specific process of preprocessing is as follows: first, for the current image P, find the M nearest neighbor images of the current image P through the scene classification extraction model; second, extract three nearest images from the M neighbors, and crop the union region containing the human-object pair for the current image P and each extracted image. Finally, splice the four regions obtained by cutting to generate a new image, and set the size of the spliced image to be the same as the size of the original image P. And according to the annotation information of the original image P, set the bounding box of the human and object of the spliced picture and other information. At the same time, combine multiple bounding boxes into a joint bounding box, and add annotation information to the spliced picture. This step increases the number of marked human-object pairs, but keeps the total number of learned image samples unchanged.

[0013] Step 2, based on the processed picture, feature extraction is performed through the convolutional neural network, and a one-dimensional feature vector is obtained through the flattening operation, denoted as input feature mapping.

[0014] The data obtained after step 1 is processed as the input of the convolutional neural network, and the output features are obtained after multiple dimension reductions. Then a one-dimensional feature vector is obtained through the flattening operation, denoted as input feature mapping. Through step-by-step dimension reduction and feature extraction, more compact and meaningful feature representations are obtained from the original image for subsequent task processing and analysis.

[0015] Step 3, based on the input feature mapping, perform human-object interaction relationship detection based on cascaded decoding and double difficult sample mining, and output the human-object interaction detection result.

[0016] Step 3.1, encoder construction based on the encoder architecture of DETR, extract the encoder features from the input feature mapping.

[0017] The encoder adopts the design of the standard encoder architecture based on DETR, and each encoder is composed of 6 layers of blocks, each layer is composed of a multi-head attention module, a feedforward network layer and a normalization layer in sequence. By processing the feature mapping, the encoder can construct higher-level feature representations of the image. This design conforms to the encoder specification of DETR and can effectively extract feature information of the image as the basis for subsequent detection tasks.

[0018] Step 3.2, construct a cascaded decoder and a double difficult sample mining architecture, and output the detection results respectively.

[0019] Two cascaded decoders are constructed for the detection of human-object pairs and the detection of interaction relations, respectively. Both of the two cascaded decoders are three-layer structures. The two decoders not only conform to the DETR decoder architecture, but also add a difficult-to-divide forward query queue mining method based on true value box movement and a difficult-to-divide forward query queue mining method based on attention mapping masking in each layer. In the human-object pair detection decoder, the learnable query Q l and the feature vector are taken as Q, K and V vectors, the visual features are processed through the operation of multi-head attention, and an updated query queue is output. The Hungarian algorithm is used to realize bipartite graph matching, the learnable query Q l is input, and a result subset closest to the true value is obtained. In the difficult-to-divide sample mining method based on attention mapping masking, the positive query Q m is obtained according to the result of bipartite graph matching, the attention scores in Q m are ranked from high to low, and the elements corresponding to the first half of the attention scores are randomly masked to realize the implicit mining of the difficult-to-divide forward query and improve the robustness of the learnable query Q l . In the difficult-to-divide sample mining method based on true value box movement, the difficult-to-divide forward query Q s is explicitly constructed according to the true value position of the human-object pair in the training image, so that the moved box only contains clues for prediction, and the robustness to the uneven distribution of spatial data is enhanced. The output queue of the last layer of the human-object pair detection decoder is output to the interaction relation decoder as prior knowledge, and the human-object interaction category is output. In the training process, the alternating joint learning strategy is adopted, and the difficult-to-divide sample mining method based on true value box movement and the difficult-to-divide sample mining method based on attention mapping masking are alternately selected for training, so that the model has higher sample mining efficiency.

[0020] Step 3.3, fuse the output of the two detection results to obtain the final human-object interaction detection result.

[0021] The outputs of the two cascaded decoders are combined, and based on the non-maximum suppression method, the repeated detection results of the same human-object interaction pair are deleted according to the overlapping degree of the detection boxes of the human-object interaction relation, and the final human-object interaction detection result is filtered out.

[0022] The beneficial effects of the present application are:

[0023] The image context consistency splicing module is used in training to obtain new learning images. The newly obtained images replace the original images, keeping the total amount of learning samples unchanged, improving the ensemble prediction ability of the model, and making the distribution of human-object interaction pairs in the image closer to the cluster center. In human-object interaction detection, the frequency distribution of human-object interaction relationship classes has a long tail, which makes the output results of the model more inclined to more common interaction classes, causing prediction bias. This module selects human-object interaction relationship blocks in the original image, crops and splices to obtain new learning samples, increases the frequency of low-frequency human-object interaction pairs, and alleviates the problem of declining recognition performance caused by uneven frequency distribution. This module uses the human-object relationship in the original image and its labeled data, and the new learning samples obtained by splicing do not need to be relabeled, solving the problem of difficult and laborious human-object pair labeling in complex scenes. The total amount of training samples is unchanged, and the rare human-object interaction pairs in a single sample are enriched, solving the problem of unbalanced training data samples.

[0024] The technology uses two cascading decoders, a human-object pair decoder and an interaction relationship decoder, to significantly improve the detection accuracy of the model for human-object interaction relationships by decoupling end-to-end human-object interaction detection into two sub-tasks. Compared to the human-object interaction relationship detection method using the same decoder, which is insufficient in capturing dependency relationships and limits the performance of the model, this technology uses two independent decoders, a human-object pair decoder and an interaction relationship decoder, for human-object pair detection and interaction relationship detection. This technology enables the model to more accurately locate human-object pairs, improves the effectiveness of interaction dependencies, and achieves high-accuracy human-object interaction relationship detection.

[0025] The technology adopts a double difficult sample mining strategy to improve the model training efficiency. The robustness and prediction accuracy of the model are improved by a difficult positive sample mining method. In the cascaded decoder part of the model, a difficult positive query queue is explicitly constructed using a true value box moving difficult sample mining method based on the true value box moving difficult sample mining method. The result query queue only contains the rough position information of the human-object pair, so that the model can better handle the influence of object position changes on human-object interaction relationship detection and improve the performance of the model. And an attention mapping based masking difficult sample mining method is used to implicitly construct a difficult positive query queue. The highest scoring part of the attention mapping is masked, so that the model only uses part of the clues in the query queue for prediction, enhancing the robustness of the learnable query. The mining of difficult positive queries effectively improves the model's ability to correctly predict human-object interaction relationship instances and enhances the model's robustness to spatially unevenly distributed data. During the training process, the invention selects an alternating joint learning strategy to alternately select the attention mapping based masking difficult sample mining method and the true value box moving difficult sample mining method based on the true value box moving difficult sample mining method for training to improve training efficiency. This strategy not only ensures the training speed, but also ensures the prediction accuracy of the model for human-object interaction relationships. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The overall flowchart of the present application is shown in the figure;

[0027] Figure 2 The structure diagram of the human-object interaction relationship detection method of the present application is shown in the figure;

[0028] Figure 3 The flowchart of the image context consistency splicing method is shown in the figure;

[0029] Figure 4 The structure diagram of the true value box moving difficult sample mining method is shown in the figure;

[0030] Figure 5 The self-attention Self-attention structure is shown in the figure;

[0031] Figure 6 The cross-attention structure is shown in the figure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application more clear and obvious, the present application will be further explained and described below in combination with the drawings.

[0033] As Figure 1 shown, a human-object interaction relationship detection method based on cascaded decoding and double difficult sample mining takes a human-object interaction pair containing image to be detected as input, and performs human-object interaction relationship detection end-to-end, specifically including the following steps:

[0034] Step 1, using the picture context consistency splicing method, the obtained picture containing the person is processed.

[0035] The flow is shown in Figure 3 A probability threshold γ is set, at the probability γ, a random number between 0 and 1 is randomly generated, and the probability value is assigned to each picture, if the probability value of the picture is greater than the probability threshold γ, the picture is preprocessed, otherwise it is not preprocessed. The picture is input to the scene classifier based on AlexNet. The classifier requires the input to be an RGB three-channel image. In the first five convolutional layers of AlexNet, the local response normalization (LRN) method is used. This method calculates the sum of squares of pixels around each position on the feature map, and then divides the current position pixel by this sum to get the normalized value of the i-th feature map at position (x, y), as shown in formula 1.

[0036]

[0037] where represents the original value of the i-th feature map at position (x, y), n represents the neighborhood size, N represents the number of feature maps, and the remaining parameters are hyperparameters. In the last three layers of fully connected layers of AlexNet, the first two layers first input the feature map into the ReLU activation function, and use Dropout to improve the generalization ability of the network, and the last fully connected layer outputs the scene category classification through softmax. The specific formulas of relu and softmax are as follows:

[0038] f(x) = max(0, x) (2)

[0039]

[0040] When processing the current input image, the first M scene category probabilities from the output are selected as neighbors. Then, 3 images are randomly selected from the M neighbors, and the union region of the marked person-object pair in each image is cropped, and finally a new image is generated by splicing the four regions. The size of the spliced image is set to the size of the original image. And according to the annotation information of the original image, set the new image obtained by splicing to the person and object bounding box information. Merge multiple bounding boxes into a joint bounding box, and add specific person-object interaction annotation information to the new image obtained by splicing.

[0041] Step 2, based on the processed picture, feature extraction is performed through the convolutional neural network, and a one-dimensional feature vector is obtained through the flattening operation, denoted as input feature mapping.

[0042] The human-object interaction relationship detection model proposed in the present application adopts ResNet50 as the backbone network, and requires input of a 3-channel RGB image x∈R3×H×W where H and W represent the height and width of the input image, respectively. After inputting the image into the backbone, the feature map where H' and W' are the height and width of the output feature map, D b is the number of channels. Then, the feature map z b will be input into a projection convolutional layer with a kernel size of 1x1 to reduce the dimension of the feature map from D b to D c . Then, a flattening operator is used to operate on the feature map to reduce the dimension to one dimension, generating flattened features

[0043] Step 3, based on the input feature map, perform cascaded decoding and double difficult sample mining human-object interaction relationship detection, output human-object interaction detection results.

[0044] Step 3.1, build an encoder

[0045] Build an encoder based on the DETR standard, each encoder consists of six layers of blocks, input the reduced dimension feature into the encoder, and use the self-attention mechanism (as shown in Figure 5 ) to generate a feature map with richer context information. In addition, position encoding is input into the encoder to complete the position information. Finally, the encoder generates a feature map with rich context information.

[0046] Step 3.2, build a cascaded decoder and double difficult sample mining method

[0047] Build two three-layer cascaded decoders, these two decoders have the same architecture, which are human-object pair decoder and interaction decoder, and each decoder layer uses several feedforward neural networks FFN for intermediate monitoring. Specifically, in addition to following the basic architecture of the decoder based on the Detection Transformer, the true value box moving difficult sample mining method and the attention mapping masking difficult sample mining method are additionally added. Each FFN consists of one or several multi-layer perceptron branches, different branches are used for specific tasks, and all queries share the same structure of FFN. In the process of feeding forward, first feed in a set of learnable queries Q l , the decoder first applies self-attention operation to the learnable query Q l , then performs cross-attention between the learnable query Q l and the one-dimensional feature, as shown in Figure 6 , the result C i is as shown in equation 4:

[0048]

[0049]

[0050] in is the normalized cross-attention map of the h-th attention head, as shown in Formula 5; T represents the number of cross-attention heads, is a linear projection matrix, Att h (·) is the similarity calculation function. l After bipartite graph matching, the positive query Q is obtained from the result of bipartite graph matching based on the attention map masked hard-to-distinguish sample mining method m , whose attention map is recorded as A m , while querying Q m Some elements of the cross attention map are masked to improve the learnable query Q l The pseudo code is as follows:

[0051]

[0052] First, initialize a random binary mask M, and M i,j ~Bernoulli(γ), obeys the Bernoulli distribution with parameter γ, each (i, j) belongs to I K M i,j Assign it to 1, otherwise assign it to 0, and map the attention to A m Update the masked feature map A after doing Hadamard product with M m , that is, implicitly constructing the hard-to-classify positive query Q m In this way, the hard positive query Q m Each query in must obtain more visual clues from the non-masked area to improve the learnable query Q l The attention map-based masked hard-to-distinguish sample mining method requires two additional parameters K and γ. Since the ultimate goal is to enhance the learnable query Q l The robustness of l Select A by the corresponding value in m The masked elements in the . First, according to the cross attention map A l The first K elements are selected and then masked with probability γ m In training, K is set to 100 and γ is set to 0.4.

[0053] In order to improve the model's ability to correctly predict instances of human-object interaction relationships, a method for mining hard-to-distinguish samples based on ground truth box movement is added to the decoder (e.g. Figure 4The hard positive query Q is constructed according to the ground truth of each labeled human-object pair in the training image s . Specifically, the bounding box of the labeled human-object pair is shifted so that only local feature cues for human-object interaction relationship detection are contained in the shifted box, effectively improving the model's ability to correctly predict human-object interaction relationship instances. For the constructed hard positive query q s ∈Q s The encoding of q

[0054] q s = L n (F p (Shift(p s ))) (6)

[0055] where p s = [x h , y h , w h , h h , x o , y o , w o , h o , x h -x o , y h -y o , w h h h , w o h o ] T The first 8 elements are the center coordinates, width and height of the human-object pair ground truth, respectively, [x h -x o , y h -y o ] represent the relative position between the two boxes, and the last two elements are the area size of the two boxes. Shift(·) represents the shift operation on the ground truth box, F p is the FFN used to project p s , L n (·) is a normalization function to ensure that the element amplitude of q s is consistent with the position embedding of the feature. During training, the IoU of the shifted box and the ground truth box ranges from 0.4 to 0.6.

[0056] However, using the true value box moving difficult sample mining method and the attention mapping based difficult sample mining method to construct difficult forward queries in training at the same time can reduce the learning efficiency and increase the training time of the model. Therefore, the application adopts an alternating joint learning strategy. Each time training alternately selects the true value box moving difficult sample mining method and the attention mapping based difficult sample mining method to construct a difficult positive query queue and realize difficult sample mining. The application inputs the learnable query and the generated difficult positive query into the decoder in turn, because the masking attention score for the difficult query in the attention mapping based difficult sample mining method takes the score of the learnable query Q l .

[0057] The model training process proposed in the application refers to the training process of query-based paired person-image interaction detection (QPIC) with image range context information. In the training stage, the Hungarian algorithm is used to match each true value and predicted value. The overall loss function of the model is shown in formula 5

[0058]

[0059] wherein L b represents the boundary box regression loss, L GIoU represents the cross overlap merging loss, L p is the interaction score loss, is the object class loss, is the target class loss, L h represents the loss for the difficult positive query, L h is specifically shown in formula (8). In addition, λ b , λ GIoU , λ p , λ o , λ a , λ h are parameters for adjusting the weight of each loss.

[0060]

[0061] Three different feedforward networks are used in the person-object pair decoder to respectively predict h box , o box , o cls The interaction decoder predicts the interaction relationship i cls through a feedforward network. The specific expression is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] wherein represents the decoding feature of the human-object pair, represents the decoding feature of the interaction.

[0067] Step 3.2, fusion of detection results

[0068] After the combination of the output results of the feedforward networks into human-object-interaction classification triplets, the scores of the triplets are obtained in combination with the outputs of the two feedforward networks, and the top K triplets are generated after being arranged in descending order. In order to further improve the accuracy of the results, the non-maximum suppression method is used to delete the repeated detection results of the same human-object interaction relationship pair, and the accurate results are further filtered out. Specifically, the overlapping degree between the pair of overlapping bounding boxes is calculated, and the most accurate result is retained according to the overlapping degree. The calculation result of the two overlapping is:

[0069]

[0070] wherein I and U are used to calculate the overlapping parts of the two boxes m and n, and γ and δ are the balance parameters used to adjust the weights of the human and the object to optimize the problem, and b h and b o represent the human detection box and the object detection box in the pair of detection boxes, respectively.

[0071] Further, as shown in Table 1, the method of the present application achieves better results compared with the two-stage methods (IDN, VCL, GGNet) and the single-stage methods (QPIC, PPDM, HOTR, AS-Net) in recent years. The experimental results in Table 1 are the mAP of the Default scene on the HICO-DET dataset, wherein Full represents the complete 600 human-object interaction class combinations, Rare represents the 138 HOI human-object interaction class combinations with less than 10 HOI training instances, and Non-Rare represents other 462 human-object interaction class non-rare sets.

[0072] The single-stage and two-stage human-object interaction relationship detection methods are constantly improving. The single-stage method has achieved similar effects to the two-stage method. Compared with both the two-stage method and the single-stage method, the method of the present application achieves the optimal performance on the detection task of the Rare human-object interaction class and also achieves the optimal performance on the detection task of the Non-Rare human-object interaction class. Since the present application achieves the optimal performance on the Rare human-object interaction class and the Non-Rare human-object interaction class, it also achieves the optimal performance on the detection task of the overall human-object interaction class.

[0073] Table 1

[0074] Method Type Year Full Rare Non-Rare IDN Two-Stage 2020 26.29 22.61 27.39 VCL Two-Stage 2020 23.63 17.21 25.55 GGNet Two-Stage 2021 28.83 22.13 30.84 PPDM One-Stage 2020 20.93 18.24 21.78 QPIC One-Stage 2021 29.07 21.85 31.23 HOTR One-Stage 2021 25.10 22.13 30.84 AS-Net One-Stage 2021 28.87 24.25 30.25 Invention One-Stage - 31.03 26.33 32.74

[0075] As shown in Table 2, the method achieves the best performance on the V-COCO dataset in terms of the metrics, and outperforms most methods in terms of the metrics, and achieves good performance for overall human-object interaction detection. In the V-COCO dataset, some human-object interaction pairs are not defined by labels. Among them, in the corresponding scenarios, the detector is required to report the case where no object exists. indicates that for the case where the object label is missing, the model ignores the detection of the object bounding box.

[0076] Table 2

[0077]

[0078] These results show that the method of the present application achieves good performance on the two datasets mentioned above, showing its superior robustness. The experimental data show that the present application effectively improves the accuracy of human-object interaction relationship detection.

Claims

1. A method for detecting person interaction relationships based on cascade decoding and dual-hard sample mining, characterized in that: The following steps are involved: Step 1: Use the image context consistency stitching method to process the acquired images containing people. The specific process is as follows: Step 1.1: Set a probability threshold γ, randomly generate a random number between 0 and 1, and assign a probability value to each image. If the probability value of the image is greater than the probability threshold γ, the image is preprocessed, otherwise no preprocessing is performed; Step 1.2: Use the image context consistency stitching method to preprocess the images in the dataset; The preprocessing process is as follows: First, for the current image P, the scene classification extraction model is used to find the M neighbor images closest to the current image P. Second, the three closest images are extracted from the M neighbors. For the current image P and each extracted image, the union region containing the person-object pair is cropped. Finally, the four cropped regions are concatenated to generate a new image. The spliced ​​image is resized to the same size as image P. Based on the annotation information of image P, the bounding box information of the person and object is set for the spliced ​​image. Step 2: Based on the processed image, feature extraction is performed through a convolutional neural network, and then a one-dimensional feature vector is obtained through a flattening operation, which is recorded as the input feature map; Step 3: Based on the input feature map, perform cascade decoding and dual-hard sample mining to detect the human-object interaction relationship and output the human-object interaction detection result. The specific process is as follows: Step 3.1: Construct an encoder based on the DETR encoder architecture and extract encoder features from the input feature map. Step 3.2: Build the cascade decoder and the dual-hard sample mining architecture, and output the detection results respectively. The specific operations are as follows: Construct two cascaded decoders, one for detecting person-object pairs and the other for detecting interaction relationships; The two cascade decoders are both three-layer structures, which conform to the DETR decoder architecture. In addition, a method for mining hard-to-distinguish samples based on moving ground truth boxes and a method for mining hard-to-distinguish samples based on masking of attention maps are added to each layer to construct a hard-to-distinguish forward query queue. In the person-object pair detection decoder, the query Q l And the feature vector is used as Q, K, V vector, and the visual features are processed through the multi-head attention operation to output a set of updated query queues; the Hungarian algorithm is used to implement bipartite graph matching, and the input can be learned query Q l , get the result subset closest to the true value; In the masked hard-to-distinguish sample mining method based on attention mapping, the positive query Q is obtained according to the result of bipartite graph matching. m , Q m The elements with the highest attention scores are ranked at the bottom, and the elements corresponding to the first half of the attention scores are randomly masked to achieve implicit mining of hard-to-separate forward queries and improve the learnable query Q. l In the method of mining hard-to-distinguish samples based on the moving ground truth box, the hard-to-distinguish positive query Q is explicitly constructed according to the ground truth position of the person-object pair in the training image. s , so that the moved box only contains clues for prediction; The output queue of the last layer of the human-object pair detection decoder is output as prior knowledge to the interaction relationship decoder, which outputs the human-object interaction category; Step 3.3: Fuse the two detection results to obtain the final human-object interaction detection result.

2. The method for detecting interactive relationships between people based on cascade decoding and double-hard sample mining according to claim 1, characterized in that: The pre-processing process also includes merging multiple bounding boxes into a joint bounding box and adding annotation information to the spliced ​​image.

3. The method for detecting interactive relationships between people based on cascade decoding and double-hard sample mining according to claim 2, characterized in that: The encoder described in step 3.1 adopts the design of the standard encoder architecture based on DETR. Each encoder consists of 6 layers of blocks, and each layer of block consists of a multi-head attention module, a feedforward network layer and a normalization layer in sequence.

4. The method for detecting person interaction relationships based on cascade decoding and double-hard sample mining according to claim 3, characterized in that: The specific process of step 3.3 is: combine the outputs of the two cascade decoders, and based on the non-maximum suppression method, delete the repeated detection results of the same human-object interaction relationship pair according to the degree of overlap of the human-object interaction relationship detection frame, and filter out the final human-object interaction detection result.

5. The method for detecting person interaction relationships based on cascade decoding and double-hard sample mining according to claim 4 is characterized in that: Step 3.2 also includes, during the training process, adopting an alternating joint learning strategy, alternatingly selecting a method for mining difficult samples based on ground truth box movement and a method for mining difficult samples based on masking of attention mapping for training.

Images