Method for detecting human interaction based on instance space correlation

Abstract

The application discloses a person interaction detection method based on instance space correlation, comprising the following steps: data collection; constructing an instance-in and instance-between person interaction detection network model, including a feature extraction part and a Transformer structure part, wherein the Transformer structure part comprises an encoder, an instance decoder and an interaction decoder, the instance decoder is a multi-layer structure, each layer comprises a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network and a bounding box regression; the instance decoder can decode multi-scale encoded features, generate person and object detection results, and be used for interaction recognition; inputting an image into the trained instance-in and instance-between person interaction detection network model, outputting the bounding boxes of the person and the object, the category of the object and the interaction category, and performing person interaction relationship detection; and the application has the characteristics of good model generalization, high instance detection accuracy and high interaction classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a person interaction detection method based on instance space association. BACKGROUND

[0002] Person interaction detection (HOI) is to detect people and objects in images or videos and explain the interaction actions between people and objects. HOI methods include two categories: two-stage methods and one-stage methods. Two-stage methods take a simple approach, which is to first use a trained object detector to detect people and objects, and then use various features to identify the interaction actions between people. The core of the two-stage method is to find the real interaction relationship between people and objects among numerous objects. Since the interaction classification in the two-stage method is usually a multi-label classification task, it often produces too many negative examples. One-stage methods usually use object detection models such as DETR or DeformableDETR as the main framework, taking advantage of their powerful object detection capabilities and long-distance dependency extraction capabilities.

[0003] In the prior art, in order to alleviate the problem of too many negative examples in the two-stage method, some scholars have migrated DETR to HOI and proposed QPIC. This method uses a single query to simultaneously detect people and objects, match people, and classify interactions, and achieves single-label classification of person interaction by focusing on the interaction area, thereby avoiding the generation of too many negative examples. At the same time, other scholars have decoupled instance detection and interaction classification and proposed AS-Net and HOTR. These two methods use interaction vectors as guidance to match the people and objects corresponding to the interaction. The above methods reduce negative examples compared to two-stage methods, but they require a long time for pre-training and fine-tuning, and their effectiveness is not obvious for complex cases of person position and person posture class combinations. In addition to the above, both one-stage and two-stage methods have the problem of being difficult to generalize to rare samples and complex samples in actual scenes. Therefore, in order to improve generalization, some studies use clip distillation semantic knowledge to jointly help scene graph generation and person interaction detection, and the generated scene graph contains fine-grained relationships that can guide person interaction detection. Some studies align interaction features to text features and use text features to help infer interaction classification. Although these methods enhance the generalization of the model by introducing additional information, the additional information requires more feature processing and feature alignment, further increasing the complexity of the model. In summary, the current person interaction detection has the problem of model generalization due to the complex combination of person position and person posture classes, and the accuracy of instance detection and interaction classification is not high. SUMMARY

[0004] The present application aims to overcome the above-mentioned shortcomings and proposes a person interaction detection method based on instance space correlation, which has good model generalization, high instance detection and interaction classification accuracy.

[0005] The present application is a person interaction detection method based on instance space correlation, wherein the specific steps of the method include:

[0006] Step 1: Data collection: collect image data with people and objects to form a data set;

[0007] Step 2: Model construction: construct an instance-in and instance-out person interaction detection network model, including a feature extraction part and a Transformer structure part;

[0008] The feature extraction part: the backbone network is a visual network Swin-Transformer Tiny, which extracts multi-scale feature maps F from the input image x through the backbone network mul The extracted feature maps can not only capture global features, but also extract features of different scales from multiple resolutions, providing diverse inputs for subsequent person detection and person interaction relationship recognition;

[0009] The Transformer structure part: including an encoder, an instance decoder and an interaction decoder, the encoder is a multi-layer structure, each layer including a multi-scale variable self-attention module and a feedforward neural network module; the encoder inputs the multi-scale feature maps F from the feature extraction part mul , and generates multi-scale high-dimensional features F e for the instance decoder and the interaction decoder;

[0010] The instance decoder is a multi-layer structure, each layer including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network and a bounding box regression; the input of the instance decoder is the instance query vector Q ins , the position vector Q pos , the instance bounding box reference position P ref and the multi-scale high-dimensional encoding feature F e , and the output is the instance feature F ins , the instance category C ins and the updated instance bounding box reference position B ins ; the instance decoder can decode multi-scale encoding features to generate person and object detection results, and is used for interaction recognition; the instance query vector Q ins includes the person query vector Q sub and the object query vector Q obj , i.e. Q ins =[Q sub ,Q obj] is used to capture the category features of instances and search the location of instances; the location vector Q pos Initialize the location vector for the machine, including the location vector Q spos of the person and the location vector Q opos of the object, that is, Q pos =[Q spos ,Q opos ], which is used to expand the diversity of the query vector; the instance bounding box reference position P ref is the center point position of the instance bounding box.

[0011] The interaction decoder is a multi-layer structure, each layer including a spatial relationship extraction module, a feature update module, and an interaction relationship extraction module; the input of the interaction decoder is the instance feature F ins , the bounding box reference B ins , and the feature atlas F deep , and the output is the interaction category C act ; the interaction decoder integrates spatial layout features, allowing interaction features to have spatial relationships between people and objects, thereby improving the spatial perception ability of the features;

[0012] Step 3: Train the model: use the dataset to train the intra-instance and inter-instance human interaction detection network model;

[0013] Step 4: Input the image into the trained intra-instance and inter-instance human interaction detection network model to output the bounding box of the person and the object, the category of the object, and the interaction category, and perform human interaction relationship detection.

[0014] The above-mentioned human interaction detection method based on instance spatial association, wherein in step 1, the dataset is a public dataset HICO-Det.

[0015] The above-mentioned human interaction detection method based on instance spatial association, wherein in step 2, the encoder is a 6-layer structure, each layer including a multi-scale variable self-attention module and a feedforward neural network module.

[0016] The above-mentioned human interaction detection method based on instance spatial association, wherein in step 2, the encoder specifically processes the following steps:

[0017] a) input multi-scale features F mul , obtain multi-scale attention through the multi-scale variable self-attention module;

[0018] b) then pass through the feedforward neural network module to realize high-dimensional enhancement, nonlinear transformation, and low-dimensional focused features, generating multi-scale high-dimensional features F e , wherein the generated F e has three scale feature atlases F low , Fmed , F deep , F e will be used for instance decoder, while F deep will be used for interactive decoder.

[0019] The above-mentioned method for detecting human interaction based on instance space correlation, wherein: in step 2, the instance decoder is a 3-layer structure, each layer including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression.

[0020] The above-mentioned method for detecting human interaction based on instance space correlation, wherein: in step 2, the specific processing steps of the instance decoder are:

[0021] a) instance query vector Q ins is added to position vector Q pos to ensure the position difference of different query features;

[0022] b) calculate self-attention query vector through multi-head self-attention module;

[0023] c) combine self-attention query vector, bounding box reference P ref , and multi-scale high-dimensional encoding feature F e together to input into multi-scale variable attention module to calculate variable cross-attention feature;

[0024] d) obtain instance feature F ins through high-dimensional enhancement, nonlinear transformation and low-dimensional focusing of feedforward neural network, and obtain object classification C obj through linear mapping at the same time;

[0025] e) F ins goes through bounding box regression and adds P ref to obtain updated instance bounding box reference position B ins , which is expressed as:

[0026] F ins = Dec ins (Q ins +Q pos , P ref , F e )

[0027] B ins =P ref +BE(F ins )

[0028] In the formula, Dec ins (*) is the calculation of instance decoder; BE(*) is the calculation of bounding box regression.

[0029] The method for detecting human interaction based on instance space correlation, wherein: in step 2, the interaction decoder has a three-layer structure, each layer including a space relation extraction, a feature update part, and an interaction relation extraction part.

[0030] The method for detecting human interaction based on instance space correlation, wherein: in step 2, the specific processing steps of the interaction decoder are as follows:

[0031] a) updating the instance bounding box reference position B ins to a spatial relation parameter, and then performing multi-layer linear mapping to align the dimension with the feature hidden layer dimension D;

[0032] b) splitting the instance feature F ins = [F sub , F obj ], and adding the human instance feature F sub and the object instance feature F obj to obtain the interaction query Q act , and then adding the interaction feature F act -1 of the previous layer to obtain the action feature RQ act , and similarly adding the spatial query and the spatial feature F lay -1 of the previous layer to obtain the spatial relation RQ lay ;

[0033] c) adding the weighted action feature RQ act and the weighted spatial relation feature RQ lay to input self-attention to obtain the coarse-grained spatial relation feature F lay , and then inputting the deep feature atlas F deep and cross-attention to obtain the fine-grained interaction feature F act ;

[0034] d) adding the interaction feature F act and the spatial relation feature F lay after linear mapping to obtain the interaction category C act .

[0035] The method for detecting human interaction based on instance space correlation, wherein: in step 3, the loss calculation of the human interaction detection network model in the training instance is as follows:

[0036] 1) object classification loss L obj : representing the difference between the predicted object classification and the true category, the object classification loss is calculated using the cross-entropy function, and the formula is,

[0037]

[0038] where N q is the number of query vectors, D is the dimension of hidden layer, C gt is the real class vector of the object, C obj is the output object classification result.

[0039] 2) Interactive action classification loss L act : represents the difference between the predicted interactive class and the real class, calculated using Focal Loss, the formula is:

[0040]

[0041] where agt is the real action class vector, γ is the adjustment factor, C act is the output interactive classification result.

[0042] 3) Instance bounding box loss L box : represents the difference between the predicted instance bounding box and the actual bounding box, calculated using L1 loss, the formula is:

[0043]

[0044] where |·| represents L1 norm, Bgt is the actual bounding box position.

[0045] 4) Intersection over union loss of instance bounding box L IoU : represents the difference between the predicted instance bounding box and the actual bounding box in terms of intersection over union, the formula is:

[0046]

[0047] where IoU(*) represents the intersection over union.

[0048] 5) Total loss function L: the weighted sum of the above losses, the formula is:

[0049] L = L IoU + L act + L obj + λL box

[0050] where λ is the adjustment factor for adjusting the bounding box loss.

[0051] The application has obvious beneficial effects compared with the prior art. As can be seen from the above scheme, the application constructs an instance-to-instance interaction detection network model in the example, which includes a feature extraction part and a Transformer structure part. The Transformer structure part includes an encoder, an instance decoder, and an interaction decoder. The instance decoder is a multi-layer structure, each layer of which includes a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression. The input of the instance decoder is an instance query vector Qins, a position vector Q pos , an instance bounding box reference position P ref , and a multi-scale high-dimensional encoded feature F e , and the output is an instance feature F ins , an instance category C ins , and an updated instance bounding box reference position B ins . The instance decoder can decode multi-scale encoded features to generate human and object detection results and be used for interaction recognition. The application uses the spatial correlation of the instance itself in the previous layer for the instance feature query in the current layer, and the queried instance feature can update the spatial position information of the current layer, gradually focusing the instance feature and the spatial position on the human-object pair, thereby improving the detection and matching capability of complex human-object combinations, and directly enhancing human-object detection and matching through the parallelism and focusing of instances. The interaction decoder is a multi-layer structure, each layer of which includes a spatial relationship extraction module, a feature update module, and an interaction relationship extraction module. The input of the interaction decoder is an instance feature F ins , a bounding box reference B ins , and a feature atlas F deep , and the output is an interaction category C act . The interaction decoder integrates spatial layout features to make the interaction features have spatial relationships between humans and objects, thereby improving the spatial perception capability of the features. The application considers the spatial information in the model, extracts the spatial position relationship between humans and objects, integrates the spatial relationship into the interaction query to realize the mutual cooperation between features, realizes the mutual antagonistic relationship between features through the jump connection of different layers, and enhances the robustness of interaction classification. The main advantages of the application are as follows:

[0052] 1) The application can realize human-object detection through the spatial connection between the front and rear layers, and make the spatial correlation relationship of humans and objects participate in the interaction classification.

[0053] 2) The application uses variable attention to realize a parallel iteration strategy to provide prior knowledge for target positioning of instance detection, and helps to improve the detection and matching accuracy of humans and objects by associating the spatial information of the front and rear layers of the instance.

[0054] 3) The present application uses the human-object space information and content features output by the instance branch to extract the spatial distribution relationship features between the human and the object, and guides the classification of human-object interaction in a skip connection manner.

[0055] In summary, the present application has the characteristics of good model generalization, high instance detection and interaction classification accuracy.

[0056] The beneficial effects of the present application are further described below through specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The flowchart of the present application is shown;

[0058] Figure 2 The structure diagram of the interaction detection network model of the present application is shown;

[0059] Figure 3 The structure diagram of the instance decoder of the present application is shown;

[0060] Figure 4 The structure diagram of the interaction decoder of the present application is shown. DETAILED DESCRIPTION

[0061] The specific embodiments, features and effects of the human interaction detection method based on instance space association according to the present application are described in detail as follows in combination with the preferred embodiments and the accompanying drawings.

[0062] Referring to Figure 1 The human interaction detection method based on instance space association of the present application, wherein the specific steps of the method include:

[0063] Step 1: Data collection part

[0064] Various human interaction detection data are collected from the public data set HICO-Det, which is image data with human, object and human-object interaction information. The HICO-Det data set includes 80 object categories and 117 action categories, a total of 600 human interaction categories.

[0065] Step 2: Model construction:

[0066] Referring to Figure 2 An intra-instance and inter-instance human interaction detection network model (ILCN) is constructed, specifically including:

[0067] 2.1 Feature extraction part

[0068] Swin-Transformer Tiny is selected as the backbone network to extract multi-scale feature maps F from the input image x mulThe extracted feature map can not only capture global features, but also extract features of different scales from multiple resolutions, providing diverse inputs for subsequent human-object detection and human interaction relationship recognition. The formula is:

[0069] F mul = Swin_T(x)

[0070] wherein x represents an input image, Swin_T(*) represents a Swin-Transformer Tiny network, F mul includes three feature maps, which respectively represent the outputs of the second stage, the third stage and the fourth stage of Swin_T.

[0071] 2.2 Query vector initialization part

[0072] An instance query vector, a position vector and a reference point coordinate are constructed, the instance query vector is used to capture the category features of the instance and search the position of the instance, the position vector is used to expand the diversity of the query vector, and the reference point coordinate is the reference position of the boundary box of the instance.

[0073] Embodiments:

[0074] According to the initial setting vector number N q and the feature hidden layer dimension D,

[0075] 1) Construct an instance query vector Q ins , the first half is the query vector Q sub of the person, and the second half is the query vector Q obj of the object, that is, Q ins = [Q sub , Q obj ];

[0076] 2) Construct a randomly initialized position vector Q pos , and Q pos = [Q spos , Q opos ];

[0077] 3) A random reference point vector Q ref is constructed.

[0078] 4) Q ref is converted into the initial instance boundary box center point P ref = [P sub , P obj ] by using a linear layer.

[0079] 2.3 Transformer structure part

[0080] The part includes an encoder, an instance decoder, and an interaction decoder. Specifically, the instance decoder represents the target detection process of people and objects, and the interaction decoder represents the interaction recognition process between people and objects. It should be noted that the encoder part and the instance decoder part are deformable DETR (Deformable-Detr) structures.

[0081] 1) Encoder:

[0082] Encode input features to generate multi-scale high-dimensional encoded features, focusing on multi-scale image information. The input is the multi-scale feature map F mul from the feature extraction part, and the generated multi-scale high-dimensional features F e will be used in the instance decoder and interaction decoder parts.

[0083] Specifically, the encoder structure has 6 layers, each including a multi-scale variable self-attention module and a feedforward neural network module.

[0084] Implementation steps:

[0085] a) Input features F mul pass through a multi-scale variable self-attention module to obtain multi-scale attention;

[0086] b) Then pass through a feedforward neural network module to realize high-dimensional enhancement, nonlinear transformation, and low-dimensional focused features;

[0087] Output F e There are 3 scale feature maps, F low , F med , and F deep . F e will be used in the instance decoder, and F deep will be used in the interaction decoder

[0088] 2) Instance decoder:

[0089] Referring to Figure 3 , decode multi-scale encoded features to generate detection results of people and objects, and for interaction recognition. The input is the instance query vector Q ins , the position vector Q pos , the instance bounding box reference position P ref , and the multi-scale high-dimensional encoded features F e , and the output is the instance feature F ins , the instance category C ins , and the updated instance bounding box reference position B ins .

[0090] Specifically, the instance decoder has 3 layers, each including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and a bounding box regression.

[0091] Implementation steps:

[0092] a) Example query vector Q ins is added to ensure the position difference of different query features; pos

[0093] b) Then calculate the self-attention query vector through the multi-head self-attention module;

[0094] c) Combine the self-attention query vector, the bounding box reference P ref and the multi-scale high-dimensional encoding feature F e , and input them into the multi-scale variable attention module to calculate the variable cross-attention feature;

[0095] d) Through the high-dimensional enhancement of the feedforward neural network, nonlinear transformation and low-dimensional focusing, the instance feature F ins is obtained, and the object classification C obj is obtained through linear mapping;

[0096] e) F ins is added to P ref to obtain the updated instance bounding box reference position B ins .

[0097] F ins = Dec ins (Q ins + Q pos , P ref , F e )

[0098] B ins = P ref + BE(F ins )

[0099] In the formula, Dec ins is an instance decoder; and BE is a bounding box regression.

[0100] For each layer of the instance decoder, since the instance query vector Q ins is the concatenation of the person query vector Q sub and the object query vector Q obj , the instance feature F ins also includes the person feature F sub and the object feature F obj ; similarly, the bounding box reference B ins = [B sub , B obj ].

[0101] After F ins and B​ins Then, they are respectively taken as the Q ins and P ref input of the next layer until the decoder output of the Nth layer is the final F N ins and B N ins for training or inference. In addition, F ins and B ins are also used as inputs for the next interactive recognition of the same layer interactive decoder part.

[0102] The embodiment allows the feature to use the spatial correlation of the instance of the previous layer itself for the instance feature query of the current layer; and the queried instance feature can update the spatial position information of the current layer, gradually focusing the instance feature and the spatial position on the human-object pairing, thereby improving the detection and matching ability of complex human-object combinations, thereby directly enhancing human-object detection and matching through parallel and focusing of instances.

[0103] 3) Interactive decoder:

[0104] Referring to Figure 4 , by integrating spatial layout features, the interactive feature has the spatial relationship between the human and the object, improving the spatial perception ability of the feature. The attention mechanism helps the feature to capture deep semantic features, and the spatial relationship and the interactive feature are mutually opposed through the jump connection mechanism to improve the robustness of the model. The input is the instance feature F ins and the bounding box reference B ins and F deep , and the output is the interactive category C act .

[0105] Specifically, the interactive decoder has 3 layers, each including a spatial relationship extraction module, a feature update module, and an interactive relationship extraction module.

[0106] Implementation steps:

[0107] a) Use spatial relationship extraction to convert the bounding box reference P ref (B ins ) into spatial relationship parameters (including the center point of the person, the relative height, the relative width, the area ratio between persons, the height-width ratio, the intersection-union ratio, and the distance), and then through multi-layer linear mapping to align the dimension with D;

[0108] b) split F ins = [F sub , F obj ], and add F sub and F obj to obtain the interactive query Q act , and then add the interactive feature F act-1 Weighted sum to obtain action features RQ act ; Similarly, the spatial query is combined with the spatial features F lay -1 Weighted sum to obtain spatial relationship RQ lay This part takes the input of the previous layer as the output of the next layer, which allows the gradient to be passed back to the shallower layer, promoting the update of the parameters of the shallow module;

[0109] c) Add the weighted action features RQ act and the weighted spatial relationship features RQ lay to input self-attention to obtain coarse-grained spatial relationship features F lay , and then input cross-attention with the deep feature atlas F deep to obtain fine-grained interaction features F act ;

[0110] d) Add the interaction features F act and the spatial relationship features F lay after linear mapping to obtain interaction categories C act

[0111] The embodiment differs from the method using additional information, which considers the spatial information within the model, extracts the spatial position relationship between people and objects, and integrates the spatial relationship into the interaction query to realize the cooperation between features. Through the jump connection between different layers, the relationship between features is realized to enhance the robustness of interaction classification.

[0112] Step 3: Train the model

[0113] Use the dataset to train the inter-instance and intra-instance human interaction detection network model; wherein the loss calculation compares the predicted human and object bounding boxes B ins , object categories C obj and interaction categories C act with their true values, including:

[0114] 1) Object classification loss L obj : represents the difference between the predicted object classification and the true category, and the object classification loss is calculated using the cross-entropy function, the formula is,

[0115]

[0116] In the formula, N q is the number of query vectors, D is the hidden layer dimension, C gt is the true category vector of the object, and C obj is the output object classification result;

[0117] 2) Interaction action classification loss Lact represents the difference between the predicted interaction category and the real category, calculated using Focal Loss, with the formula:

[0118]

[0119] wherein a gt is the real action category vector, γ is an adjustment factor, C act is the output interaction classification result;

[0120] 3) Instance bounding box loss L box represents the difference between the predicted instance bounding box and the actual bounding box, calculated using L1 loss, with the formula:

[0121]

[0122] wherein |·| represents L1 norm, B gt is the actual bounding box position;

[0123] 4) Intersection over union loss of instance bounding box L IoU represents the difference between the predicted instance bounding box and the actual bounding box in terms of intersection over union, with the formula:

[0124]

[0125] wherein IoU(*) represents the intersection over union;

[0126] 5) Total loss function L: weighted sum of the above losses, with the formula:

[0127] L = L IoU + L act + L obj + λL box

[0128] wherein λ is an adjustment factor for adjusting the bounding box loss.

[0129] Step 4: Interaction relationship detection

[0130] Input the image into the trained instance-in and instance-out human-object interaction detection network model, output the bounding box of people and objects, the category of objects, and the interaction category, and perform human interaction relationship detection.

[0131] Performance analysis

[0132] To verify the effectiveness of the present application, a test experiment was conducted on two NVIDIA-A40 GPUs, and the experiment showed that significant performance improvement was achieved. On the benchmark human interaction detection dataset HICO-DET, compared with the current optimal method PDN (2024) which does not use additional information, the method provided by the present application has a relative improvement of 0.62 in average precision mAP. Compared with the IA-L (2024) method which uses additional information, an improvement of 1.16 is obtained. It can be seen that the human interaction detection method provided by the present application can fully utilize visual features for detection and classification.

[0133] In addition, under the premise of the same backbone network and without using COCO pre-training model parameters, comparative test experiments were conducted on the method provided by the present application and other Deformable-Detr-based methods, and Table 1 shows the comparative results of ILCN and representative Deformable-Detr-based models.

[0134] Table 1 Comparative results of ILCN and representative Deformable-Detr-based models

[0135]

[0136] As can be seen from Table 1, with only 100 training rounds, ILCN has the highest mean average precision (mAP) of 30.76, which is increased by 2.31 compared with QAHOI, about 8.1%; increased by 0.82 compared with FGAHOI, about 2.7%; the parameter amount of ILCN is 40.37M, and the frame per second (FPS) is 8.33, which is slightly higher than that of QAHOI, but is significantly improved compared with that of FGAHOI; at the same time, ILCN has less floating point operations per second (FLOPS). Therefore, according to the above analysis, ILCN has higher detection accuracy compared with the representative method, and has competitive performance in detection efficiency.

[0137] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification of the above embodiment according to the technical essence of the present application, without departing from the technical solution of the present application, still belongs to the scope of the technical solution of the present application.

Claims

1. A method for detecting character interactions based on instance space association, characterized in that: The specific steps of this method include: Step 1: Data Collection: Collect image data of people and objects to form a dataset; Step 2: Model Building: Construct a network model for detecting intra- and inter-instance human interactions, including feature extraction and Transformer structure. The feature extraction part uses a Swin-Transformer Tiny visual network as its backbone network. This part extracts multi-scale feature maps F from the input image x through the backbone network. mul The extracted feature maps can not only capture global features, but also extract features at different scales from multiple resolutions, providing diverse inputs for subsequent person detection and recognition of person interaction relationships; The Transformer structure includes an encoder, an instance decoder, and an interactive decoder. The encoder is a multi-layer structure, with each layer including a multi-scale variable self-attention module and a feedforward neural network module. The encoder's input is a multi-scale feature map F from the feature extraction part. mul The generated multi-scale high-dimensional features F e Used for instance decoders and interactive decoders; The instance decoder has a multi-layer structure, with each layer including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and bounding box regression; the input of the instance decoder is the instance query vector Q. ins Position vector Q pos Instance bounding box reference position P ref and multi-scale high-dimensional encoding features F e The output is the instance feature F. ins Instance category C ins And the updated instance bounding box reference position B ins This instance decoder can decode multi-scale encoded features, generate detection results for people and objects, and use them for interactive recognition; the instance query vector Q ins Including the query vector Q of people sub The query vector Q of the object obj Q ins =[Q sub Q obj The location vector Q is used to capture the category features of instances and search for the location of instances. pos Initialize the machine's position vector, including the human's position vector Q. spos The position vector Q of the object opos Q pos =[Q spos Q opos ], used to expand the diversity of query vectors; the instance bounding box reference position P ref This refers to the location of the center point of the instance bounding box; The interactive decoder has a multi-layer structure, with each layer including a spatial relationship extraction module, a feature update module, and an interaction relationship extraction module; the input of the interactive decoder is the instance feature F. ins Bounding box reference B ins Feature map F deep The output is the interaction category C. act This interactive decoder incorporates spatial layout features, enabling interactive features to possess spatial relationships between people and objects, thereby improving the spatial perception capabilities of the features. Step 3: Train the model: Train the network model for detecting intra- and inter-instance character interactions using the dataset; Step 4: Input the image into the trained intra- and inter-instance human interaction detection network model, and output the bounding boxes of people and objects, the category of objects and the interaction category to detect human interaction relationships.

2. The character interaction detection method based on instance space association as described in claim 1, characterized in that: In step 1, the dataset is the public dataset HICO-Det.

3. The character interaction detection method based on instance space association as described in claim 1, characterized in that: In step 2, the encoder has a 6-layer structure, with each layer including a multi-scale variable self-attention module and a feedforward neural network module.

4. The character interaction detection method based on instance space association as described in claim 3, characterized in that: In step 2, the encoder performs the following specific processing steps: a) Input multi-scale features F mul Multi-scale attention is obtained through a multi-scale variable self-attention module; b) Then, through a feedforward neural network module, high-dimensional enhancement, nonlinear transformation, and low-dimensional focusing are achieved to generate multi-scale high-dimensional features F. e The generated F e There are three scales of feature maps F low F med F deep F e It will be used for the instance decoder, while F deep This is used for interactive decoders.

5. The character interaction detection method based on instance space association as described in claim 1, characterized in that: In step 2, the instance decoder has a 3-layer structure, each layer including a multi-head self-attention module, a multi-scale variable attention module, a feedforward neural network, and bounding box regression.

6. The character interaction detection method based on instance space association as described in claim 5, characterized in that: In step 2, the specific processing steps of the instance decoder are as follows: a) Instance query vector Q ins With position vector Q pos Adding them together ensures that the positions of different query features are distinguishable. b) The self-attention query vector is calculated using the multi-head self-attention module; c) Combining self-attention query vectors and bounding box references P ref Multi-scale high-dimensional coding features F e The common input is a multi-scale variable attention module, which calculates variable cross-attention features; d) After high-dimensional enhancement, nonlinear transformation, and low-dimensional focusing through a feedforward neural network, the instance features F are obtained. ins Simultaneously, object classification C is obtained through linear mapping. obj ; e)F ins After bounding box regression, and P ref Add to obtain the updated instance bounding box reference position B ins , is represented as: F ins =Dec ins (Q ins +Q pos ,P ref ,F e ) B ins =P ref +BE(F ins ) In the formula, Dec ins (*) is calculated by the instance decoder; BE(*) is calculated by the bounding box regressor.

7. The character interaction detection method based on instance space association as described in claim 1, characterized in that: In step 2, the interactive decoder has a three-layer structure, each layer including a spatial relationship extraction part, a feature update part, and an interaction relationship extraction part.

8. The character interaction detection method based on instance space association as described in claim 7, characterized in that: In step 2, the specific processing steps of the interactive decoder are as follows: a) Use spatial relationships to extract the reference position B of the updated instance bounding box. ins The parameters are transformed into spatial relationship parameters, and then the dimensions are aligned with the dimension D of the hidden feature layer through multiple linear mappings. b) Segmentation instance features F ins =[F sub F obj ], and the instance features F of people sub Instance characteristics F of objects obj Add them together to get the interactive query Q act Then, interact with the previous layer's features F. act -1 Weighted summation to obtain action features RQ act Similarly, spatial queries are linked to the spatial features F of the previous layer. lay -1 Weighted summation yields spatial relation RQ lay ; c) Weighted action feature RQ act Weighted spatial relation features RQ lay Adding the inputs to self-attention yields coarse-grained spatial relation features F. lay Then, combined with the deep feature map F deep Input cross-attention to obtain fine-grained interaction features F act ; d) Interaction features F act Spatial relationship characteristics F lay After linear mapping, the interaction category C is obtained by addition. act .

9. The character interaction detection method based on instance space association as described in claim 1, characterized in that: In step 3, the loss of the network model for detecting character interactions within training instances and between training instances is calculated as follows: 1) Object classification loss L obj This represents the difference between the predicted object category and the true object category. The cross-entropy function is used to calculate the object classification loss, and the formula is as follows: In the formula, N q Where D is the number of query vectors, C is the dimension of the hidden layer, and D is the dimension of the hidden layer. gt C is the true class vector of the object. obj The output is the object classification result; 2) Interactive action classification loss L act : Represents the difference between the predicted interaction category and the true category, calculated using FocalLoss, with the formula: In the formula, a gt Let C be the actual action category vector, γ be the adjustment factor, and C be the value of C. act The output is the interactive classification result; 3) Instance bounding box loss L box : Represents the difference between the predicted instance bounding box and the actual bounding box, calculated using L1 loss, with the formula: In the formula, |·| represents the L1 norm, B gt This represents the actual bounding box location; 4) Intersection over Union (IoU) loss L of instance bounding boxes IoU : This represents the difference between the intersection-union ratio (IU) of the predicted instance bounding box and the actual bounding box, expressed by the formula: In the formula, IoU(*) represents the intersection-union ratio; 5) Total loss function L: The total loss is calculated by weighting and summing the losses mentioned above. The formula is as follows: L=L IoU +L act +L obj +λL box In the formula, λ is the adjustment factor for adjusting the bounding box loss.

Images