Character interaction detection method based on content query and position query

Abstract

The invention is suitable for the technical field of character interaction detection, and provides a character interaction detection method based on content query and position query, which comprises the following steps of: firstly, extracting image features through a backbone network, detecting a person and an object instance by using a target detector DETR, then pairing the person and the object instance, and obtaining a matched person and object instance; the method comprises the following steps: extracting appearance, space and object category semantic features, calculating interactive scores, selecting interactive suggestions, deeply fusing multi-modal features, embedding generated contents and positions, transmitting the multi-modal features into a Transform decoder for cross attention calculation to obtain interactive features, finally inputting the interactive features into a classifier, and combining the confidence scores to obtain a classification result. And outputting the score of the character interaction triad. According to the method, through deep fusion of multi-modal features and a cross attention mechanism, the detection precision of character and object interaction is effectively improved, and the character interaction relationship in a complex scene can be accurately identified.

Description

Technical Field

[0001] The present application belongs to the technical field of human interaction detection, and in particular, relates to a human interaction detection method based on content query and location query. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, the processing and analysis of image and video data has become increasingly important. Especially in multiple application scenarios such as autonomous driving, security monitoring, and social network analysis, how to understand human behavior in images and the interaction between humans and objects has become a crucial issue. Traditional computer vision technology mainly focuses on object detection and image classification, but these methods often fail to fully capture the complex human activities and behavior patterns in images. Therefore, human-object interaction detection (HOI) has emerged. Human-object interaction detection aims to understand and analyze human activities by detecting humans, objects, and their interactions in images or videos. Specifically, it provides a structured way for computers to accurately detect complex interactive behaviors in different scenarios by organizing people (including human bounding boxes), objects (including object bounding boxes and categories), and interaction categories into triplets (<people, actions, objects>).

[0003] Methods for human interaction detection are mainly divided into two schools: two-stage methods and one-stage methods. The two-stage method first uses a pre-trained object detection framework to detect humans and objects in the image, then pairs the detected humans and objects, and inputs the features of these pairs into the interaction classification network to determine the interaction relationship between them; while the one-stage method allows the network to predict human interaction triplets end-to-end by introducing a new HOI intermediary (human interaction point, human joint box, etc.). Recently, due to the success of Transformer in the field of computer vision, Transformer-based human interaction detection methods have become mainstream. They use the powerful expression ability of Transformer to extract rich and high-quality features from images and achieve excellent performance. The most advanced human interaction detection method is the two-stage method based on Transformer, which has faster convergence speed and better detection accuracy than the one-stage method, but it also has some shortcomings: it needs to calculate a large number of non-interactive redundant human pairs, which greatly affects the detection efficiency; it only uses the human and object features obtained from the object detector and the manually designed spatial features for interaction recognition, and cannot use some key visual context clues to recognize some complex interactions. Summary of the invention

[0004] The embodiment of the present application provides a method for detecting human interaction based on content query and location query, which can solve the problem that the current two-stage human interaction detection cannot fully utilize visual context clues to identify complex interaction relationships.

[0005] In the first aspect, an embodiment of the present application provides a method for detecting human interaction based on content query and position query, comprising the following steps: S1, inputting an image to be detected into a backbone network to obtain image visual features, flattening the image into multiple one-dimensional vectors, enhancing the image through sinusoidal position coding, and passing the vectors into a target detector DETR to obtain all human instances and object instances; S2, pairing human instances and object instances one by one to obtain multiple human pairs, extracting multimodal features of the human pairs, wherein the multimodal features include appearance features, spatial features, and object category semantic features, connecting the appearance features, the spatial features, and the object category semantic features and passing them into a multi-layer perceptron to obtain an interactivity score for each human pair, and passing the score with the highest value into a multi-layer perceptron. Multiple character pairs with high probability are determined as multiple character interaction proposals; S3, the multimodal features of the character pairs extracted in S2 are deeply fused to obtain the content embedding of the character interaction proposal, and the center point coordinates of the person frame and the object frame are used to construct the position embedding of the character interaction proposal; S4, the content embedding and position embedding of multiple character interaction proposals are passed into the preset Transformer decoder to generate content queries and position queries, and decoupled cross-attention calculations are performed with image features and image position encoding to obtain interaction features; S5, the interaction features are passed into the action classifier to obtain the score of each action category, and the confidence scores of the person instance and object instance obtained in S1 are combined to obtain the score of the character interaction triplet.

[0006] In a possible implementation manner of the first aspect, the method of S1 is specifically: S101, the original features of the image to be detected Input CNN backbone network to get feature map ,in is a set of real numbers, W and H are the width and height of the image to be detected, W' and H' are the width and height of the feature map, and D is the number of channels of the feature map; S102, calculate the two-dimensional position encoding of the feature map F: The position code corresponding to the (x,y) position of the feature map F is , flatten the feature map F to get the feature sequence , let the feature sequence Add the corresponding position code to get the feature sequence with position information ; S103, feature sequence Input the target detector DETR to obtain all the human instances and object instances in the image to be detected. Each instance is represented as , where b represents the bounding box of the instance, c represents the instance category, s represents the confidence score, and x represents the feature of the instance output by the DETR decoder.

[0007] Optionally, in another possible implementation manner of the first aspect, the method in S2 is specifically: S201, pairing the detected person instances and object instances one by one to obtain a plurality of person pairs; S202: For each person pair, calculate its appearance features ,in and are the features of person instances and object instances respectively; S203, for each person pair, calculate its spatial feature: in is the center point coordinate of the human frame, is the center point coordinate of the object frame, For the width and height of the human frame, are the width and height of the object frame, is the intersection-over-intersection ratio of the person frame and the object frame, Represents the absolute value operation; S204: For each person pair, calculate the semantic features of the object category : Construct a text prompt in the form of a photo of a / an[object], where [object] is the object category, pass the text prompt into CLIP's text encoder to obtain text embedding, and use the text embedding as the semantic feature of the object category; S205, the appearance features, spatial features and object category semantic features of the character pairs are concatenated and passed into a multi-layer perceptron, and the interactivity scores are output by a Sigmoid activation function, and the multiple character pairs with the highest scores are used as character interaction proposals.

[0008] Optionally, in another possible implementation of the first aspect, the method in S3 is specifically as follows: S301: For each character interaction proposal, use the multimodal features of the character pair extracted in S2 to build content embedding , the method is as follows: in The appearance characteristics of the representative character pair, The spatial characteristics of the representative character pair, represents the object category semantic features of person pairs, and MBF stands for multi-branch fusion module; S302: For each person interaction proposal, construct a position embedding using the center point coordinates of the person frame and the object frame. , the method is as follows: in is the coordinate of the center point of the human frame, are the coordinates of the center point of the object frame, The role of is to map the scalar to the sine embedding space, which is calculated as , where i = 1, 2, 3, ..., d / 2.

[0009] Optionally, in another possible implementation of the first aspect, the method in S4 is specifically as follows: S401, passing the content embedding and position embedding of multiple character interaction proposals into a preset Transformer decoder, where the preset Transformer decoder consists of 8 decoder layers; S402: The current decoder layer receives multiple embeddings output by the previous decoder layer as the input sequence E, and obtains the updated input sequence E after self-attention calculation, residual connection, and layer normalization. ; S403: Input sequence As content query, the position embedding obtained by S3 is used as the position query, and multiple content queries and position queries are concatenated to obtain the query for cross attention calculation , that is, the i-th query The calculation method is: ,in represents the i-th content query, represents the i-th position query; connect each feature point in the feature sequence F' obtained by S1 with the position code corresponding to the feature point to obtain the key of cross attention , i.e. the i-th key The calculation method is: ,in represents the i-th feature point in the feature sequence F', Represents the position code corresponding to the i-th feature point; calculate the attention weight a of q for k by the dot multiplication method, that is, the i-th query For the jth key The attention weight The calculation method is: ; Take the feature sequence F' as the cross-attention value v, and get the output of the cross-attention part through weighted summation of attention weights. Then, after residual connection, layer normalization and feedforward neural network, get the output embedding of the decoder layer and input it to the next decoder layer. The output embedding of the last decoder layer is used as the interaction feature of the character interaction proposal.

[0010] Optionally, in another possible implementation of the first aspect, the method in S5 is specifically as follows: S501, passing the interaction features of each character interaction proposal into the action classifier, which is a multi-layer perceptron equipped with a Sigmoid activation function, to obtain the score of each action category, and combining the confidence scores of the person instance and the object instance obtained in S1 to obtain the score s of the character interaction triplet, which is calculated as follows: in is the confidence score of the person instance, is the confidence score of the object instance, is the score of the action category.

[0011] Beneficial effect: In the technical solution of the present application, the image to be detected is first input into the backbone network, the visual features of the image are extracted, and the image is enhanced by sinusoidal position coding, and then input into the target detector DETR to obtain human and object instances, and then the human and object instances are paired, and their appearance features, spatial features and object category semantic features are extracted, and the interactivity scores are input into the multi-layer perceptron to calculate the interactivity scores, and multiple character pairs with the highest scores are selected as interaction proposals, and then the multimodal features of the character pairs are deeply fused to generate content embedding, and the position embedding is constructed based on the center coordinates of the human frame and the object frame, and then the content embedding and the position embedding are input into the Transformer decoder to generate content query and position query, and cross-attention calculation is performed to obtain interaction features, and finally the interaction features are input into the action classifier, and the image features and confidence scores are combined to calculate and output the scores of the character interaction triples. Therefore, by deeply fusing multimodal features and cross-attention mechanisms, the detection accuracy of the interaction between people and objects is effectively improved, and the interaction relationship between people in complex scenes can be accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1It is a flowchart of a method for detecting human interaction based on content query and location query provided in one embodiment of the present application. DETAILED DESCRIPTION

[0014] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0015] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0016] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0017] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0018] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0019] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0020] The following describes in detail the method for detecting human interaction based on content query and location query provided by the present application with reference to the accompanying drawings.

[0021] Figure 1 A flow chart of a method for detecting human interaction based on content query and location query provided in an embodiment of the present application is shown.

[0022] like Figure 1 As shown, the human interaction detection method based on content query and location query includes the following steps: S1. Input the image to be detected into the backbone network to obtain the image visual features, flatten them into multiple one-dimensional vectors, enhance them through sinusoidal position encoding, and input them into the target detector DETR to obtain all the human instances and object instances; Furthermore, in the embodiment of the present application, the method of S1 is specifically as follows: S101, the original features of the image to be detected Input CNN backbone network to get feature map ,in is a set of real numbers, W and H are the width and height of the image to be detected, W' and H' are the width and height of the feature map, and D is the number of channels of the feature map; S102, calculate the two-dimensional position encoding of the feature map F: The position code corresponding to the (x,y) position of the feature map F is , flatten the feature map F to get the feature sequence , let the feature sequence Add the corresponding position code to get the feature sequence with position information ; S103, the feature sequence F'' Input the target detector DETR to obtain all the human instances and object instances in the image to be detected. Each instance is represented as , where b represents the bounding box of the instance, c represents the instance category, s represents the confidence score, and x represents the feature of the instance output by the DETR decoder.

[0023] S2. Pairing the person instances and the object instances one by one to obtain multiple person pairs, extracting multimodal features of the person pairs, the multimodal features including appearance features, spatial features and object category semantic features, connecting the appearance features, spatial features and object category semantic features and passing them into a multi-layer perceptron to obtain the interactivity score of each person pair, and determining the multiple person pairs with the highest scores as multiple person interaction proposals; Furthermore, in the embodiment of the present application, the method of S2 is specifically as follows: S201, pairing the detected person instances and object instances one by one to obtain a plurality of person pairs; S202: For each person pair, calculate its appearance features ,in and are the features of person instances and object instances respectively; S203, for each person pair, calculate its spatial feature: in is the center point coordinate of the human frame, is the center point coordinate of the object frame, For the width and height of the human frame, are the width and height of the object frame, is the intersection-over-intersection ratio of the person frame and the object frame, Represents the absolute value operation; S204: For each person pair, calculate the semantic features of the object category : Construct a text prompt in the form of a photo of a / an[object], where [object] is the object category, pass the text prompt into CLIP's text encoder to obtain text embedding, and use the text embedding as the semantic feature of the object category; S205, the appearance features, spatial features and object category semantic features of the character pairs are concatenated and passed into a multi-layer perceptron, and the interactivity scores are output by a Sigmoid activation function, and the multiple character pairs with the highest scores are used as character interaction proposals.

[0024] It should be noted that during training, the loss function of interactive prediction is , where FL represents the Focal Loss function, Indicates whether the i-th person pair is a positive sample with interaction (for a person pair, if the intersection and union ratio of its person frame and object frame with the person frame and object frame in the ground truth are both greater than 0.5, then the person pair is considered a positive sample, otherwise it is a negative sample). represents the predicted interaction score of the i-th character pair.

[0025] S3, deeply fuse the multimodal features of the person pairs extracted in S2 to obtain the content embedding of the person interaction proposal, and use the center point coordinates of the person frame and the object frame to construct the position embedding of the person interaction proposal; Furthermore, in the embodiment of the present application, the method of S3 is specifically as follows: S301: For each character interaction proposal, use the multimodal features of the character pair extracted in S2 to build content embedding , the method is as follows: in The appearance characteristics of the representative character pair, The spatial characteristics of the representative character pair, represents the object category semantic features of person pairs, and MBF stands for multi-branch fusion module; S302: For each person interaction proposal, construct a position embedding using the center point coordinates of the person frame and the object frame. , the method is as follows: in is the coordinate of the center point of the human frame, are the coordinates of the center point of the object frame, The role of is to map the scalar to the sine embedding space, which is calculated as , where i = 1, 2, 3, ..., d / 2.

[0026] S4, passing the content embedding and position embedding of multiple character interaction proposals into the preset Transformer decoder, generating content query and position query, and performing decoupled cross-attention calculation with image features and image position encoding to obtain interaction features; Furthermore, in the embodiment of the present application, the method of S4 is specifically as follows: S401, passing the content embedding and position embedding of multiple character interaction proposals into a preset Transformer decoder, where the preset Transformer decoder consists of 8 decoder layers; S402: The current decoder layer receives multiple embeddings output by the previous decoder layer as the input sequence E, and obtains the updated input sequence E after self-attention calculation, residual connection, and layer normalization. ; S403: Input sequence As content query, the position embedding obtained by S3 is used as the position query, and multiple content queries and position queries are concatenated to obtain the query for cross attention calculation , that is, the i-th query The calculation method is: ,in represents the i-th content query, represents the i-th position query; connect each feature point in the feature sequence F' obtained by S1 with the position code corresponding to the feature point to obtain the key of cross attention , i.e. the i-th key The calculation method is: ,in represents the i-th feature point in the feature sequence F', Represents the position code corresponding to the i-th feature point; calculate the attention weight a of q for k by the dot multiplication method, that is, the i-th query For the jth key The attention weight The calculation method is: ; Take the feature sequence F' as the cross-attention value v, and get the output of the cross-attention part through weighted summation of attention weights. Then, after residual connection, layer normalization and feedforward neural network, get the output embedding of the decoder layer and input it to the next decoder layer. The output embedding of the last decoder layer is used as the interaction feature of the character interaction proposal.

[0027] It should be noted that from the i-th query For the jth key The attention weight Calculation method: It can be seen that by decoupling the query into content query and location query, the attention mechanism can be guided from different aspects: content query can obtain relevant information from the image based on the person features, while location query can explicitly guide the model to focus on the image features near the center points of the person box and object box, which is more conducive to discovering visual context clues.

[0028] S5: pass the interaction features into the action classifier to obtain the score of each action category, and combine the confidence scores of the person instance and object instance obtained in S1 to obtain the score of the person interaction triplet.

[0029] Furthermore, in the embodiment of the present application, the method of S5 is specifically as follows: S501, passing the interaction features of each character interaction proposal into the action classifier, which is a multi-layer perceptron equipped with a Sigmoid activation function, to obtain the score of each action category, and combining the confidence scores of the person instance and the object instance obtained in S1 to obtain the score s of the character interaction triplet, which is calculated as follows: in is the confidence score of the person instance, is the confidence score of the object instance, is the score of the action category.

[0030] As a possible implementation, during training, the total loss function of the entire network is The calculation method is as follows:

[0031] Where C represents the number of interaction categories, N represents the number of character interaction proposals, Indicates whether the label corresponding to the i-th character interaction proposal has the c-th interaction category, The probability that the cth interaction category exists for the predicted i-th character interaction proposal, is the loss of interactivity prediction in S2.

[0032] The human interaction detection method based on content query and position query provided by the present application first inputs the image to be detected into the backbone network, extracts the visual features of the image, and enhances it through sinusoidal position coding, and then inputs it into the target detector DETR to obtain human and object instances, then pairs the human and object instances, extracts their appearance features, spatial features and object category semantic features, and inputs them into the multi-layer perceptron to calculate the interactivity score, and selects multiple human pairs with the highest scores as interaction proposals, then deeply fuses the multimodal features of the human pairs, generates content embedding, and constructs position embedding based on the center coordinates of the human frame and the object frame, then inputs the content embedding and position embedding into the Transformer decoder, generates content query and position query, and performs cross-attention calculation to obtain interaction features, and finally inputs the interaction features into the action classifier, combines the image features and the confidence score, and calculates and outputs the score of the human interaction triplet. Thus, by deeply fusing multimodal features and cross-attention mechanism, the detection accuracy of human-object interaction is effectively improved, and the human interaction relationship in complex scenes can be accurately identified.

[0033] Some of the core codes of the human interaction detection method based on content query and location query in this application are as follows: (1) Calculate the spatial features of person pairs: def compute_spatial_features( boxes_1: List[Tensor], boxes_2: List[Tensor], eps: float = 1e-10 ) ->Tensor: features = [] for b1, b2, shape in zip(boxes_1, boxes_2): c1_x = (b1[:, 0] + b1[:, 2]) / 2; c1_y = (b1[:, 1] + b1[:, 3]) / 2 c2_x = (b2[:, 0] + b2[:, 2]) / 2; c2_y = (b2[:, 1] + b2[:, 3]) / 2 b1_w = b1[:, 2] - b1[:, 0]; b1_h = b1[:, 3] - b1[:, 1] b2_w = b2[:, 2] - b2[:, 0]; b2_h = b2[:, 3] - b2[:, 1] d_x = torch.abs(c2_x - c1_x) / (b1_w + eps) d_y = torch.abs(c2_y - c1_y) / (b1_h + eps) iou = torch.diag(box_ops.box_iou(b1, b2)) f = torch.stack( d_x, d_y, b2_w * b2_h / (b1_w * b1_h + eps), b1_w / (b1_h + eps), b2_w / (b2_h + eps), iou , 1) features.append(f) return torch.cat(features) (2) Calculate the CLIP text embeddings corresponding to each object category: with torch.no_grad(): device = "cuda" if torch.cuda.is_available() else "cpu" clip_model = 'ViT-B / 32' self.clip_model, preprocess = clip.load(clip_model, device=device) obj_text_inputs = torch.cat([clip.tokenize(obj_text[1]) for obj_text in obj_text_label]) self.obj_text_embedding = self.clip_model.encode_text(obj_text_inputs.to(device)).float() del clip_model (3) Calculate the content embedding of character interaction proposals: # mbf and mbf_as are two different multi-branch fusion modules content_embed = self.mbf( sem_features[x_keep, y_keep], self.mbf_as( torch.cat([embeds[x_keep], embeds[y_keep]], dim=1), spatial_features[x_keep, y_keep] ) ) (4) Calculate the position embedding of the character interaction proposal: def compute_c_pe(boxes, image_size): bx_norm = boxes / image_size[[1, 0, 1, 0]] bx_c = (bx_norm[:, :2] + bx_norm[:, 2:]) / 2 c_pe = compute_sinusoidal_pe(bx_c[:, None], 10000).squeeze(1) return c_pe c_pe = compute_c_pe(boxes, image_size) position_embed = torch.add(c_pe[x_keep], c_pe[y_keep]).unsqueeze(1) (5) Cross-attention calculation for decoupling content query and location query: # tgt is the embedding output of the self-attention part q_content = self.ca_qcontent_proj(tgt) # memory is the image feature k_content = self.ca_kcontent_proj(memory) v = self.ca_v_proj(memory) num_queries, bs, n_model = q_content.shape hw, _, _ = k_content.shape # pos is the position encoding of the image k_pos = self.ca_kpos_proj(pos) q = q_content k = k_content q = q.view(num_queries, bs, self.nhead, n_model / / self.nhead) # query_sine_embed is the position embedding of the human interaction proposal query_sine_embed = self.ca_qpos_sine_proj(query_sine_embed) query_sine_embed = query_sine_embed.view(num_queries, bs, self.nhead,n_model / / self.nhead) q = torch.cat([q, query_sine_embed], dim=3).view(num_queries, bs, n_model * 2) k = k.view(hw, bs, self.nhead, n_model / / self.nhead) k_pos = k_pos.view(hw, bs, self.nhead, n_model / / self.nhead) k = torch.cat([k, k_pos], dim=3).view(hw, bs, n_model * 2) tgt2 = self.cross_attn(query=q, key=k, value=v, attn_mask=memory_mask, key_padding_mask=memory_key_padding_mask)[0]

[0034] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0035] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting human interaction based on content query and location query, characterized in that: The steps include: S1. Input the image to be detected into the backbone network to obtain the image visual features, flatten it into multiple one-dimensional vectors, enhance it through sinusoidal position encoding, and input it into the target detector DETR to obtain all human instances and object instances; S2. Pairing the person instances and the object instances one by one to obtain multiple person pairs, extracting multimodal features of the person pairs, wherein the multimodal features include appearance features, spatial features, and object category semantic features, connecting the appearance features, the spatial features, and the object category semantic features, and transmitting them to a multi-layer perceptron to obtain an interactivity score for each person pair, and determining the multiple person pairs with the highest scores as multiple person interaction proposals; S3, deeply fuse the multimodal features of the person pairs extracted in S2 to obtain the content embedding of the person interaction proposal, and use the center point coordinates of the person frame and the object frame to construct the position embedding of the person interaction proposal; S4, passing the content embedding and position embedding of multiple character interaction proposals into the preset Transformer decoder, generating content query and position query, and performing decoupled cross-attention calculation with image features and image position encoding to obtain interaction features; S5: pass the interaction features into the action classifier to obtain the score of each action category, and combine the confidence scores of the person instance and object instance obtained in S1 to obtain the score of the person interaction triplet.

2. The method according to claim 1, characterized in that The method of S1 is specifically as follows: S101, the original features of the image to be detected Input CNN backbone network to get feature map ,in is a set of real numbers, W and H are the width and height of the image to be detected, W' and H' are the width and height of the feature map, and D is the number of channels of the feature map; S102, calculate the two-dimensional position encoding of the feature map F: The position code corresponding to the (x,y) position of the feature map F is , flatten the feature map F to get the feature sequence , let the feature sequence Add the corresponding position code to get the feature sequence with position information ; S103, the feature sequence Input the target detector DETR to obtain all the human instances and object instances in the image to be detected. Each instance is represented as , where b represents the bounding box of the instance, c represents the instance category, s represents the confidence score, and x represents the feature of the instance output by the DETR decoder.

3. The method according to claim 1, characterized in that The method of S2 is specifically as follows: S201, pairing the detected person instances and object instances one by one to obtain a plurality of person pairs; S202: For each person pair, calculate its appearance features ,in and are the features of person instances and object instances respectively; S203, for each person pair, calculate its spatial feature: in is the center point coordinate of the human frame, is the center point coordinate of the object frame, For the width and height of the human frame, are the width and height of the object frame, is the intersection-over-union ratio of the person frame and the object frame, Represents the absolute value operation; S204: For each person pair, calculate the semantic features of the object category : Construct a text prompt in the form of a photo of a / an [object], where [object] is the object category, pass the text prompt into CLIP's text encoder to obtain text embedding, and use the text embedding as the semantic feature of the object category; S205, the appearance features, spatial features and object category semantic features of the character pairs are concatenated and passed into a multi-layer perceptron, and the interactivity scores are output by a Sigmoid activation function, and the multiple character pairs with the highest scores are used as character interaction proposals.

4. The method according to claim 1, characterized in that: The S3 method is as follows: S301: For each character interaction proposal, use the multimodal features of the character pair extracted in S2 to build content embedding , the method is as follows: in The appearance characteristics of the representative character pair, The spatial characteristics of the representative character pair, represents the object category semantic features of person pairs, and MBF stands for multi-branch fusion module; S302: For each person interaction proposal, construct a position embedding using the center point coordinates of the person frame and the object frame. , the method is as follows: in is the coordinate of the center point of the human frame, are the coordinates of the center point of the object frame, The role of is to map the scalar to the sine embedding space, which is calculated as , where i = 1, 2, 3, ..., d / 2.

5. The method according to claim 2, characterized in that: The method of S4 is as follows: S401, passing the content embedding and position embedding of multiple character interaction proposals into a preset Transformer decoder, wherein the preset Transformer decoder is composed of 8 decoder layers; S402: The current decoder layer receives multiple embeddings output by the previous decoder layer as the input sequence E, and obtains the updated input sequence E after self-attention calculation, residual connection, and layer normalization. ; S403: Input sequence As content query, the position embedding obtained by S3 is used as the position query, and multiple content queries and position queries are concatenated to obtain the query for cross attention calculation , that is, the i-th query The calculation method is: ,in represents the i-th content query, represents the i-th position query; connect each feature point in the feature sequence F' obtained by S1 with the position code corresponding to the feature point to obtain the key of cross attention , i.e. the i-th key The calculation method is: ,in represents the i-th feature point in the feature sequence F', Represents the position code corresponding to the i-th feature point; calculate the attention weight a of q for k by the dot multiplication method, that is, the i-th query For the jth key The attention weight The calculation method is: ; Take the feature sequence F' as the cross-attention value v, and get the output of the cross-attention part through weighted summation of attention weights. Then, after residual connection, layer normalization and feedforward neural network, get the output embedding of the decoder layer and input it to the next decoder layer. The output embedding of the last decoder layer is used as the interaction feature of the character interaction proposal.

6. The method according to claim 1, characterized in that The method of S5 is as follows: S501, passing the interaction features of each character interaction proposal into the action classifier, which is a multi-layer perceptron equipped with a Sigmoid activation function, to obtain the score of each action category, and combining the confidence scores of the person instance and the object instance obtained in S1 to obtain the score s of the character interaction triplet, which is calculated as follows: in is the confidence score of the person instance, is the confidence score of the object instance, is the score of the action category.

Images