Figure interaction detection method based on instance space association

Abstract

The invention discloses a figure interaction detection method based on instance space association. The method comprises the following steps: collecting data; according to the method, a character interaction detection network model in instances and between instances is constructed, a feature extraction part and a Transform structure part are included, the Transform structure part comprises an encoder, an instance decoder and an interaction decoder, the instance decoder is of a multi-layer structure, and the interaction decoder is of a multi-layer structure. Each layer comprises a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network and bounding box regression; the instance decoder can decode the multi-scale coding features, generate detection results of people and objects and be used for interactive recognition; and inputting the images into the trained instances and the character interaction detection network model between the instances, outputting bounding boxes of people and objects, categories of the objects and interaction categories, and carrying out character interaction relationship detection. The method has the advantages of being good in model generalization and high in instance detection and interactive classification accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting human interaction based on instance space association. Background Art

[0002] Human interaction detection (HOI) is to detect people and objects in images or videos and explain the interactive actions between people and objects. It is gradually applied in visual question answering, behavior recognition, image retrieval and other fields. HOI methods include two categories: two-stage methods and one-stage methods. The two-stage method adopts a simple idea, that is, first use the trained target detector to realize human-object detection, and then use various features to identify the interactive actions between people. The core of the two-stage method is to find the human-object pairing relationship that really exists in the numerous targets. Since the interaction classification in the two-stage method is usually a multi-label classification task, it often generates too many negative examples. The one-stage method usually uses the target detection model DETR or DeformableDETR as the main framework, relying on their powerful target detection capabilities and the ability to extract long-distance dependencies.

[0003] In the prior art, in order to alleviate the problem of too many negative examples in the two-stage method, some scholars migrated DETR to HOI and proposed QPIC. This method uses a single query to simultaneously realize human-object detection, human matching and interaction classification. By focusing on the interaction area, single-label classification of human interaction is realized to avoid generating too many negative examples. At the same time, other scholars decoupled instance detection and interaction classification and proposed AS-Net and HOTR. These two methods use interaction vectors as guidance to match people and objects corresponding to the interaction. The above methods reduce negative examples compared with the two-stage method, but they require a long time for pre-training and fine-tuning, and the effect is not obvious for complex combinations of human position and human posture classes. In addition to the above, both the one-stage and two-stage methods have the problem of difficulty in generalizing the model to sparse samples and complex samples in actual scenes. Therefore, in order to improve generalization, some studies have used clip to distill semantic knowledge to jointly help scene graph generation and human interaction detection, and the fine-grained relationships contained in the generated scene graph can guide the detection of human interaction. Some studies align interaction features to text features and use text features to help reason about interaction classification. Although these methods enhance the generalization of the model by introducing additional information, the additional information requires more feature processing and feature alignment, which further increases the complexity of the model. In short, the current human interaction detection has model generalization problems due to the complex combination of human position and human posture classes, and the accuracy of instance detection and interaction classification is not high. Summary of the invention

[0004] The purpose of the present invention is to overcome the above shortcomings and propose a method for detecting human interaction based on instance space association, which has good model generalization and high accuracy of instance detection and interaction classification.

[0005] The present invention provides a method for detecting human interaction based on instance space association, wherein: the method specifically comprises the following steps:

[0006] Step 1: Data collection: collect image data with people and objects to form a data set;

[0007] Step 2: Model construction: Construct the intra-instance and inter-instance person interaction detection network model, including feature extraction part and Transformer structure part;

[0008] The feature extraction part: The backbone network is the visual network Swin-Transformer Tiny. This part extracts multi-scale feature maps F from the input image x through the backbone network. mul The extracted feature maps can not only capture global features, but also extract features of different scales from multiple resolutions, providing diverse inputs for subsequent person detection and person interaction relationship recognition;

[0009] The Transformer structure includes an encoder, an instance decoder, and an interactive decoder. The encoder is a multi-layer structure, each layer of which includes a multi-scale variable self-attention module and a feedforward neural network module. The encoder inputs the multi-scale feature map F from the feature extraction part. mul , the generated multi-scale high-dimensional features F e For instance decoder and interaction decoder;

[0010] The instance decoder is a multi-layer structure, each layer includes a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression; the input of the instance decoder is the instance query vector Q ins , position vector Q pos , instance bounding box reference position P ref and multi-scale high-dimensional encoding features F e , the output is the instance feature F ins , Instance Type C ins and the updated instance bounding box reference position B ins ; The instance decoder can decode multi-scale encoding features, generate detection results of people and objects, and use them for interactive recognition; the instance query vector Q ins , including the query vector Q of the person sub The query vector Q of the object obj , that is, Q ins =[Q sub ,Q obj], used to capture the category features of the instance and search for the location of the instance; the position vector Q pos Initialize the position vector for the machine, including the position vector Q of the person spos and the position vector Q of the object opos , i.e. Q pos =[Q spos ,Q opos ], used to expand the diversity of the query vector; the instance bounding box reference position P ref is the center point of the instance bounding box.

[0011] The interactive decoder is a multi-layer structure, each layer includes a spatial relationship extraction module, a feature update module, and an interactive relationship extraction module; the interactive decoder input is the instance feature F ins , Bounding box reference B ins , feature map F deep , the output is the interaction category C act ; The interactive decoder integrates spatial layout features to allow interactive features to have the spatial relationship between people and objects, thus improving the spatial perception ability of features;

[0012] Step 3: training the model: using the dataset to train the intra-instance and inter-instance person interaction detection network model;

[0013] Step 4: Input the image into the trained intra-instance and inter-instance person interaction detection network model, output the bounding boxes of people and objects, the categories of objects and the interaction categories, and perform person interaction relationship detection.

[0014] The above-mentioned method for detecting human interaction based on instance space association, wherein: in step 1, the data set is the public data set HICO-Det.

[0015] The above-mentioned person interaction detection method based on instance space association, wherein: in step 2, the encoder has a 6-layer structure, each layer includes a multi-scale variable self-attention module and a feedforward neural network module.

[0016] The above-mentioned method for detecting human interaction based on instance space association, wherein: in step 2, the encoder specifically processes the following steps:

[0017] a) Input multi-scale features F mul , obtain multi-scale attention through a multi-scale variable self-attention module;

[0018] b) Then the feedforward neural network module is used to realize the features of high-dimensional enhancement, nonlinear transformation and low-dimensional focusing, generating multi-scale high-dimensional features F e , where the generated F e There are three scale feature maps F low 、Fmed 、F deep , F e will be used for the instance decoder, and F deep It is used for interactive decoders.

[0019] The above-mentioned method for detecting human interaction based on instance space association, wherein: in step 2, the instance decoder is a three-layer structure, each layer includes a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression.

[0020] The above-mentioned method for detecting human interaction based on instance space association, wherein: in step 2, the specific processing steps of the instance decoder are:

[0021] a) Example query vector Q ins With the position vector Q pos Add together to ensure the positional distinction of different query features;

[0022] b) Calculate the self-attention query vector through the multi-head self-attention module;

[0023] c) Combine the self-attention query vector and the bounding box reference P ref , multi-scale high-dimensional encoding features F e , jointly input the multi-scale variable attention module to calculate the variable cross-attention features;

[0024] d) After high-dimensional enhancement, nonlinear transformation and low-dimensional focusing of the feedforward neural network, the instance feature F is obtained ins , and obtain the object classification C through linear mapping obj ;

[0025] e)F ins After bounding box regression, and P ref Add to get the updated instance bounding box reference position B ins , expressed as:

[0026] F ins =Dec ins (Q ins +Q pos , P ref ,F e )

[0027] B ins =P ref +BE(F ins )

[0028] In the formula, Dec ins (*) is the instance decoder calculation; BE(*) is the bounding box regressor calculation.

[0029] The above-mentioned method for detecting human interaction based on instance spatial association, wherein: in step 2, the interaction decoder has a three-layer structure, each layer includes a spatial relationship extraction part, a feature updating part, and an interaction relationship extraction part.

[0030] The above-mentioned method for detecting human interaction based on instance space association, wherein: in step 2, the specific processing steps of the interaction decoder are:

[0031] a) Use spatial relationship extraction to update the instance bounding box reference position B ins Convert it into spatial relationship parameters, and then align the dimension with the feature hidden layer dimension D through multi-layer linear mapping;

[0032] b) Segmentation instance feature F ins =[F sub , F obj ], and the instance feature F sub The instance feature F of the object obj Add together to get the interactive query Q act , and then interact with the previous layer’s feature F act -1 Weighted summation to obtain action feature RQ act , similarly, the spatial query is combined with the spatial feature F lay -1 Weighted summation to obtain spatial relationship RQ lay ;

[0033] c) The weighted action feature RQ act and weighted spatial relationship feature RQ lay Add the input self-attention to get the coarse-grained spatial relationship feature F lay , and then with the deep feature map F deep Input cross attention to obtain fine-grained interaction features F act ;

[0034] d) Interaction feature F act and spatial relationship feature F lay After linear mapping, the interaction category C is obtained by adding act .

[0035] The above-mentioned method for detecting human interaction based on instance space association, wherein: in step 3, the loss of the human interaction detection network model within the training instance and between instances is calculated as:

[0036] 1) Object classification loss L obj : Represents the difference between the predicted object classification and the true category, and uses the cross entropy function to calculate the object classification loss. The formula is,

[0037]

[0038] Where N q is the number of query vectors, D is the hidden layer dimension, C gt is the true category vector of the object, C obj The output object classification result;

[0039] 2) Interaction action classification loss L act : Represents the difference between the predicted interaction category and the true category, calculated using Focal Loss, the formula is:

[0040]

[0041] In the formula, agt is the real action category vector, γ is the adjustment factor, and C act is the output interactive classification result;

[0042] 3) Instance Bounding Box Loss L box : represents the difference between the predicted instance bounding box and the actual bounding box, calculated using L1 loss, the formula is:

[0043]

[0044] In the formula, |·| represents the L1 norm, and Bgt is the actual bounding box position;

[0045] 4) Intersection-over-Union loss of instance bounding box L IoU : represents the difference between the intersection of the predicted instance bounding box and the actual bounding box, and the formula is:

[0046]

[0047] Where, IoU(*) represents the intersection over union ratio;

[0048] 5) Total loss function L: The weighted sum of the above losses is:

[0049] L=L IoU +L act +L obj +λL box

[0050] Where λ is the adjustment factor for adjusting the bounding box loss.

[0051] Compared with the prior art, the present invention has obvious beneficial effects. It can be seen from the above scheme that the present invention constructs a network model for detecting human interactions within and between instances, including a feature extraction part and a Transformer structure part. The Transformer structure part includes an encoder, an instance decoder, and an interaction decoder. The instance decoder is a multi-layer structure, each layer including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression; the input of the instance decoder is the instance query vector Qins, the position vector Q pos , instance bounding box reference position P ref and multi-scale high-dimensional encoding features F e , the output is the instance feature F ins , Instance Type C ins and the updated instance bounding box reference position B ins ; The instance decoder can decode multi-scale coded features, generate detection results for people and objects, and use them for interactive recognition. The present invention allows features to use the spatial association of the previous layer instance itself for instance feature query of the current layer; and the queried instance features can update the spatial position information of the current layer, gradually allowing the instance features and spatial positions to focus on the pairing of people and objects, thereby improving the detection and matching capabilities of complex people-object combinations, thereby directly enhancing people-object detection and matching through the parallelization and focusing of instances. The interactive decoder is a multi-layer structure, each layer includes a spatial relationship extraction module, a feature update module, and an interactive relationship extraction module; the interactive decoder input is the instance feature F ins , Bounding box reference B ins , feature map F deep , the output is the interaction category C act ; The interactive decoder integrates spatial layout features to allow interactive features to have spatial relationships between people and objects, thereby improving the spatial perception of features. The present invention takes into account the spatial information in the model, extracts the spatial position relationship between people and objects, and integrates the spatial relationship into interactive queries to achieve mutual cooperation between features. By skipping the outputs of different layers, the relationship between features is achieved to enhance the robustness of interactive classification. The main advantages of the present invention are:

[0052] 1) The present invention can use the spatial connection between the front and back layers to realize human-object detection, and allow the spatial association relationship between people and objects to participate in interactive classification.

[0053] 2) The present invention uses variable attention to implement a parallel iteration strategy to provide prior knowledge of target positioning for instance detection, and helps improve the detection and matching accuracy of people and objects by associating the spatial information of the front and back layers of the instance.

[0054] 3) The present invention utilizes the person-object spatial information and content features output by the instance branch to extract the spatial distribution relationship features between people and objects, and guides the classification of human-object interactions in a jump connection manner.

[0055] In summary, the present invention has the characteristics of good model generalization, high accuracy of instance detection and interactive classification.

[0056] The beneficial effects of the present invention are further illustrated below through specific implementation modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a schematic diagram of the process of the present invention;

[0058] Figure 2 It is a schematic diagram of the structure of the interactive detection network model of the present invention;

[0059] Figure 3 is a schematic diagram of an example decoder structure of the present invention;

[0060] Figure 4 It is a schematic diagram of the structure of the interactive decoder of the present invention. DETAILED DESCRIPTION

[0061] The specific implementation methods, features and effects of the person interaction detection method based on instance space association proposed in the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0062] See also Figure 1 The method for detecting human interaction based on instance space association of the present invention comprises the following specific steps:

[0063] Step 1: Data Collection Part

[0064] Various human interaction detection data are collected from the public dataset HICO-Det, which is image data with information about people, objects, and interactions between people. The HICO-Det dataset includes 80 object categories and 117 action categories, for a total of 600 human interaction categories.

[0065] Step 2: Model building:

[0066] See also Figure 2 , build the intra-instance and inter-instance person interaction detection network model (ILCN), which includes:

[0067] 2.1 Feature extraction part

[0068] Choose Swin-Transformer Tiny as the backbone network to extract multi-scale feature maps F from the input image x mulThe extracted feature map can not only capture global features, but also extract features of different scales from multiple resolutions, providing diverse inputs for subsequent human-object detection and human interaction relationship recognition. The formula is:

[0069] F mul =Swin_T(x)

[0070] Where x represents the input image, Swin_T(*) represents the Swin-Transformer Tiny network, and F mul It includes 3 feature maps, which represent the second, third and fourth stage outputs of Swin_T respectively.

[0071] 2.2 Query vector initialization part

[0072] An instance query vector, a position vector and reference point coordinates are constructed. The instance query vector is used to capture the category features of the instance and the location of the search instance. The position vector is used to expand the diversity of the query vector. The reference point coordinates are the reference position of the bounding box of the instance.

[0073] Implementation plan:

[0074] According to the initial setting of the number of vectors N q and the feature hidden layer dimension D,

[0075] 1) Construct the instance query vector Q ins , the first half is the query vector Q of the person sub , the latter part is the query vector Q of the object obj , i.e. Q ins =[Q sub ,Q obj ];

[0076] 2) Construct a random initialization position vector Q pos , the same Q pos =[Q spos ,Q opos ];

[0077] 3) Construct a random reference point vector Q ref ;

[0078] 4) Use the linear layer to transform Q ref Transformed to the initial instance bounding box center point P ref =[P sub ,P obj ];

[0079] 2.3 Transformer structure

[0080] This part includes an encoder, an instance decoder, and an interaction decoder. Specifically, the instance decoder represents the target detection process of people and objects, and the interaction decoder represents the interactive recognition process between people and objects. It should be noted that the encoder part and the instance decoder part are deformable DETR (Deformable-Detr) structures.

[0081] 1) Encoder:

[0082] Encode input features to generate multi-scale high-dimensional encoded features, focusing on multi-scale image information. Input the multi-scale feature map F from the feature extraction part mul , the generated multi-scale high-dimensional features F e Will be used in both instance decoder and interaction decoder parts.

[0083] Specifically, the encoder structure has 6 layers, and each layer includes a multi-scale variable self-attention module and a feedforward neural network module.

[0084] Implementation steps:

[0085] a) Input feature F mul Multi-scale attention is obtained through a multi-scale variable self-attention module;

[0086] b) The features of high-dimensional enhancement, nonlinear transformation and low-dimensional focusing are realized through the feedforward neural network module;

[0087] Output F e There are three scale feature maps, F low , F med and F deep . F e will be used for the instance decoder, and F deep For interactive decoder

[0088] 2) Example decoder:

[0089] See also Figure 3 , decodes multi-scale encoding features, generates detection results of people and objects, and is used for interactive recognition. The input is the instance query vector Q ins , position vector Q pos , instance bounding box reference position P ref and multi-scale high-dimensional encoding features F e , the output is the instance feature F ins , Instance Type C ins and the updated instance bounding box reference position B ins .

[0090] Specifically, the instance decoder has 3 layers, each of which includes a multi-head self-attention module, a multi-scale variable attention module, a feed-forward neural network, and bounding box regression.

[0091] Implementation steps:

[0092] a) Example query vector Q ins With the position vector Q pos Add together to ensure the positional distinction of different query features;

[0093] b) Then the self-attention query vector is calculated through the multi-head self-attention module;

[0094] c) Combine the self-attention query vector and the bounding box reference P ref And the multi-scale high-dimensional encoding feature F e , jointly input the multi-scale variable attention module to calculate the variable cross-attention features;

[0095] d) After high-dimensional enhancement, nonlinear transformation and low-dimensional focusing of the feedforward neural network, the instance feature F is obtained ins , and obtain the object classification C through linear mapping obj ;

[0096] e)F ins After bounding box regression, and P ref Add to get the updated instance bounding box reference position B ins .

[0097] F ins =Dec ins (Q ins +Q pos ,P ref ,F e )

[0098] B ins =P ref +BE(F ins )

[0099] In the formula, Dec ins is instance decoder; BE is bounding box regression.

[0100] For each layer of instance decoder, since the instance query vector Q ins The query vector Q for a person sub The query vector Q of the object obj The concatenation of , so the instance feature F ins Also includes human characteristics F sub Characteristics of Japanese objects obj ; Similarly, the bounding box reference B ins =[B sub , B obj ].

[0101] After obtaining F ins and Bins Afterwards, they are treated as the Q ins and P ref Input until the decoder of the Nth layer outputs the final F N ins and B N ins For training or inference. In addition, F ins and B ins It is also used as the input of the interaction decoder part of the same layer for subsequent interaction recognition.

[0102] This implementation allows features to utilize the spatial associations of instances in the previous layer for instance feature queries in the current layer; and the queried instance features can update the spatial position information of the current layer, gradually allowing instance features and spatial positions to focus on human-object pairings, thereby improving the detection and matching capabilities of complex human-object combinations, thereby directly enhancing human-object detection and matching through parallelization and focusing of instances.

[0103] 3) Interactive decoder:

[0104] See also Figure 4 By integrating spatial layout features, the interactive features have the spatial relationship between people and objects, which improves the spatial perception ability of the features. The attention mechanism helps the features capture deep semantic features, and the skip connection mechanism allows the spatial relationship and interactive features to compete with each other, improving the robustness of the model. The input is the instance feature F ins and the bounding box reference B ins and F deep , the output is the interaction category C act .

[0105] Specifically, the interactive decoder has three layers, each of which includes a spatial relationship extraction module, a feature update module, and an interactive relationship extraction module.

[0106] Implementation steps:

[0107] a) Using spatial relationship extraction to refer to the bounding box P ref (B ins ) is converted into spatial relationship parameters (including the center point, relative height, relative width of the characters, as well as the area ratio, aspect ratio, intersection ratio and distance between the characters), and then the dimensions are aligned with D through multi-layer linear mapping;

[0108] b) Split F ins =[F sub , F obj ], and F sub With F obj Add together to get the interactive query Q act , and then interact with the previous layer’s feature F act-1 Weighted summation to obtain action feature RQ act ; Similarly, the spatial query is combined with the spatial feature F of the previous layer lay -1 Weighted summation to obtain spatial relationship RQ lay This part uses the input of the previous layer as the output of the next layer, allowing the gradient to be transferred back to a shallower position, promoting the update of shallow module parameters;

[0109] c) The weighted action feature RQ act and weighted spatial relationship feature RQ lay Add the input self-attention to get the coarse-grained spatial relationship feature F lay , and then with the deep feature map F deep Input cross attention to obtain fine-grained interaction features F act ;

[0110] d) Interaction feature F act and spatial relationship feature F lay After linear mapping, the interaction category C is obtained by adding act

[0111] This embodiment is different from the method of using additional information. This embodiment takes into account the spatial information within the model, extracts the spatial position relationship between people and objects, and integrates the spatial relationship into the interactive query to achieve mutual cooperation between features. The output of different layers is jump-connected to achieve the antagonistic relationship between features, so as to enhance the robustness of interactive classification.

[0112] Step 3: Train the model

[0113] The dataset is used to train the intra-instance and inter-instance person interaction detection network model; wherein the loss calculation is to predict the bounding box B of the person and the object ins , object category C obj and interaction category C act Comparison losses with their true values ​​include:

[0114] 1) Object classification loss L obj : Represents the difference between the predicted object classification and the true category, and uses the cross entropy function to calculate the object classification loss. The formula is,

[0115]

[0116] Where N q is the number of query vectors, D is the hidden layer dimension, C gt is the true category vector of the object, C obj The output object classification result;

[0117] 2) Interaction action classification loss Lact : Represents the difference between the predicted interaction category and the true category, calculated using Focal Loss, the formula is:

[0118]

[0119] In the formula, a gt is the real action category vector, γ is the adjustment factor, C act is the output interactive classification result;

[0120] 3) Instance Bounding Box Loss L box : represents the difference between the predicted instance bounding box and the actual bounding box, calculated using L1 loss, the formula is:

[0121]

[0122] In the formula, |·| represents the L1 norm, B gt is the actual bounding box position;

[0123] 4) Intersection-over-Union loss of instance bounding box L IoU : represents the difference between the intersection of the predicted instance bounding box and the actual bounding box, and the formula is:

[0124]

[0125] Where, IoU(*) represents the intersection over union ratio;

[0126] 5) Total loss function L: The weighted sum of the above losses is:

[0127] L=L IoU +L act +L obj +λL box

[0128] Where λ is the adjustment factor for adjusting the bounding box loss.

[0129] Step 4: Interaction Detection

[0130] The image is input into the trained intra-instance and inter-instance person interaction detection network model, and the bounding boxes of people and objects, object categories and interaction categories are output to detect the person interaction relationship.

[0131] Performance Analysis

[0132] In order to verify the effectiveness of the present invention, test experiments were carried out on two NVIDIA-A40 GPUs, and the experiments showed that significant performance improvements were achieved. On the benchmark human interaction detection dataset HICO-DET, compared with the current optimal method PDN (2024) that does not use additional information, the average precision mAP provided by the present application is relatively improved by 0.62. Compared with the IA-L (2024) method that uses additional information, an improvement of 1.16 was achieved. This shows that the human interaction detection method provided by the present application can make full use of visual features for detection and classification.

[0133] In addition, under the premise of the same backbone network and without using the COCO pre-trained model parameters, a comparative test experiment was conducted on the method provided in this application and other Deformable-Detr-based methods. Table 1 shows the comparison results of ILCN and representative Deformable-Detr-based models.

[0134] Table 1 Comparison results between ILCN and representative Deformable-Detr-based models

[0135]

[0136] As shown in Table 1, ILCN has the highest mean average precision (mAP) of 30.76 after only 100 training rounds, which is 2.31 more than QAHOI, an increase of about 8.1%; 0.82 more than FGAHOI, an increase of about 2.7%; ILCN has 40.37M parameters and 8.33 frames per second (FPS), which is slightly higher than QAHOI, but significantly better than FGAHOI; ILCN has fewer floating point operations per second (FLOPS). Therefore, based on the above analysis, ILCN has higher detection accuracy than representative methods, and has competitive performance in detection efficiency.

[0137] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A method for detecting human interaction based on instance spatial association, characterized in that: The specific steps of the method include: Step 1: Data collection: collect image data with people and objects to form a data set; Step 2: Model construction: Construct the intra-instance and inter-instance person interaction detection network model, including feature extraction part and Transformer structure part; The feature extraction part: The backbone network is the visual network Swin-Transformer Tiny. This part extracts multi-scale feature maps F from the input image x through the backbone network. mul The extracted feature maps can not only capture global features, but also extract features of different scales from multiple resolutions, providing diverse inputs for subsequent person detection and person interaction relationship recognition; The Transformer structure includes an encoder, an instance decoder, and an interactive decoder. The encoder is a multi-layer structure, each layer of which includes a multi-scale variable self-attention module and a feedforward neural network module. The encoder inputs the multi-scale feature map F from the feature extraction part. mul , the generated multi-scale high-dimensional features F e For instance decoder and interaction decoder; The instance decoder is a multi-layer structure, each layer includes a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression; the input of the instance decoder is the instance query vector Q ins , position vector Q pos , instance bounding box reference position P ref and multi-scale high-dimensional encoding features F e , the output is the instance feature F ins , Instance Type C ins and the updated instance bounding box reference position B ins ; The instance decoder can decode multi-scale encoding features, generate detection results of people and objects, and use them for interactive recognition; the instance query vector Q ins , including the query vector Q of the person sub The query vector Q of the object obj , i.e. Q ins =[Q sub ,Q obj ], used to capture the category features of the instance and search for the location of the instance; the position vector Q pos Initialize the position vector for the machine, including the position vector Q of the person spos and the position vector Q of the object opos , i.e. Q pos =[Q spos ,Q opos ], used to expand the diversity of the query vector; the instance bounding box reference position P ref is the center point of the instance bounding box. The interactive decoder is a multi-layer structure, each layer includes a spatial relationship extraction module, a feature update module, and an interactive relationship extraction module; the interactive decoder input is the instance feature F ins , Bounding box reference B ins , feature map F deep , the output is the interaction category C act ; The interactive decoder integrates spatial layout features to allow interactive features to have the spatial relationship between people and objects, thus improving the spatial perception ability of features; Step 3: training the model: using the dataset to train the intra-instance and inter-instance person interaction detection network model; Step 4: Input the image into the trained intra-instance and inter-instance person interaction detection network model, output the bounding boxes of people and objects, the categories of objects and the interaction categories, and perform person interaction relationship detection.

2. The method for detecting human interaction based on instance spatial association according to claim 1, characterized in that: In step 1, the dataset is the public dataset HICO-Det.

3. The method for detecting human interaction based on instance spatial association according to claim 1, characterized in that: In step 2, the encoder has a 6-layer structure, each layer including a multi-scale variable self-attention module and a feedforward neural network module.

4. The method for detecting human interaction based on instance spatial association as claimed in claim 3, characterized in that: In step 2, the encoder specifically processes the following steps: a) Input multi-scale features F mul , obtain multi-scale attention through a multi-scale variable self-attention module; b) Then the feedforward neural network module is used to realize the features of high-dimensional enhancement, nonlinear transformation and low-dimensional focusing to generate multi-scale high-dimensional features F e , where the generated F e There are three scale feature maps F low 、F med 、F deep , F e will be used for the instance decoder, and F deep It is used for interactive decoders.

5. The method for detecting human interaction based on instance spatial association according to claim 1, characterized in that: In step 2, the instance decoder has a three-layer structure, each layer including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression.

6. The method for detecting human interaction based on instance spatial association according to claim 5, characterized in that: In step 2, the specific processing steps of the example decoder are: a) Example query vector Q ins With the position vector Q pos Add together to ensure the positional distinction of different query features; b) Calculate the self-attention query vector through the multi-head self-attention module; c) Combine the self-attention query vector and the bounding box reference P ref , multi-scale high-dimensional encoding features F e , jointly input the multi-scale variable attention module to calculate the variable cross-attention features; d) After high-dimensional enhancement, nonlinear transformation and low-dimensional focusing of the feedforward neural network, the instance feature F is obtained ins , and obtain the object classification C through linear mapping obj ; e)F ins After bounding box regression, and P ref Add to get the updated instance bounding box reference position B ins , expressed as: F ins =Dec ins (Q ins +Q pos ,P ref ,F e ) B ins =P ref +BE(F ins ) In the formula, Dec ins (*) is the instance decoder calculation; BE(*) is the bounding box regressor calculation.

7. The method for detecting human interaction based on instance spatial association according to claim 1, characterized in that: In step 2, the interactive decoder has a three-layer structure, each layer includes a spatial relationship extraction part, a feature updating part, and an interactive relationship extraction part.

8. The method for detecting human interaction based on instance spatial association according to claim 7, characterized in that: In step 2, the specific processing steps of the interactive decoder are: a) Use spatial relationship extraction to update the instance bounding box reference position B ins Convert it into spatial relationship parameters, and then align the dimension with the feature hidden layer dimension D through multi-layer linear mapping; b) Segmentation instance feature F ins =[F sub , F obj ], and the instance feature F sub The instance feature F obj Add together to get the interactive query Q act , and then interact with the previous layer’s feature F act -1 Weighted summation to obtain action feature RQ act , similarly, the spatial query is combined with the spatial feature F lay -1 Weighted summation to obtain spatial relationship RQ lay ; c) The weighted action feature RQ act and weighted spatial relationship feature RQ lay Add the input self-attention to get the coarse-grained spatial relationship feature F lay , and then with the deep feature map F deep Input cross attention to obtain fine-grained interaction features F act ; d) Interaction feature F act and spatial relationship feature F lay After linear mapping, the interaction category C is obtained by adding act .

9. The method for detecting human interaction based on instance spatial association according to claim 1, characterized in that: In step 3, the loss of the person interaction detection network model within the training instance and between instances is calculated as: 1) Object classification loss L obj : Represents the difference between the predicted object classification and the true category, and uses the cross entropy function to calculate the object classification loss. The formula is, Where N q is the number of query vectors, D is the hidden layer dimension, C gt is the true category vector of the object, C obj The output object classification result; 2) Interaction action classification loss L act : Represents the difference between the predicted interaction category and the true category, calculated using FocalLoss, the formula is: In the formula, a gt is the real action category vector, γ is the adjustment factor, C act is the output interactive classification result; 3) Instance bounding box loss L box : represents the difference between the predicted instance bounding box and the actual bounding box, calculated using L1 loss, the formula is: In the formula, |·| represents the L1 norm, B gt is the actual bounding box position; 4) Intersection-over-Union loss of instance bounding box L IoU : represents the difference between the predicted instance bounding box and the actual bounding box. The formula is: Where, IoU(*) represents the intersection over union ratio; 5) Total loss function L: The weighted sum of the above losses is: L=L IoU +L act +L obj +λL box Where λ is the adjustment factor for adjusting the bounding box loss.

Images