Human interaction detection method based on content query and location query

Abstract

The present application is applicable to the field of human interaction detection technology, and provides a human interaction detection method based on content query and location query, comprising: first extracting image features through a backbone network, and detecting human and object instances with a target detector DETR, then pairing human and object instances, extracting appearance, space, and object category semantic features and calculating interactivity scores, selecting interaction proposals, and then deeply fusing multimodal features to generate content and position embeddings, which are passed into the Transformer decoder for cross-attention calculation to obtain interaction features, and finally inputting the interaction features into a classifier, combining them with confidence scores, and outputting the scores of human interaction triples. This method effectively improves the detection accuracy of human-object interactions by deeply fusing multimodal features and cross-attention mechanisms, and can accurately identify human interaction relationships in complex scenes.

Description

Technical Field

[0001] The present application belongs to the technical field of human interaction detection, and in particular relates to a human interaction detection method based on content query and location query. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, the processing and analysis of image and video data has become increasingly important. Understanding human behavior and interactions between humans and objects in images has become a crucial issue, particularly in diverse application scenarios such as autonomous driving, security monitoring, and social network analysis. Traditional computer vision techniques primarily focus on object detection and image classification, but these methods often fail to fully capture the complex human activities and behavior patterns in images. Consequently, human-object interaction (HOI) detection has emerged. Human-object interaction detection aims to understand and analyze human activities by detecting humans, objects, and their interactions in images or videos. Specifically, it organizes humans (including their bounding boxes), objects (including their bounding boxes and categories), and interaction categories into triplets (<person, action, object>), providing computers with a structured approach to accurately detect complex interactions in diverse scenarios.

[0003] Methods for human interaction detection are mainly divided into two schools: two-stage and one-stage. Two-stage methods first use a pre-trained object detection framework to detect people and objects in an image. They then pair the detected people and objects and input these paired features into an interaction classification network to determine their interaction relationship. One-stage methods, on the other hand, introduce a new HOI intermediary (such as human interaction points or joint bounding boxes) to allow the network to predict human interaction triplets end-to-end. Recently, due to the success of the Transformer in computer vision, Transformer-based human interaction detection methods have become mainstream. Leveraging the powerful expressive power of the Transformer, these methods extract rich, high-quality features from images, achieving excellent performance. The most advanced human interaction detection method is the Transformer-based two-stage method, which has faster convergence and better detection accuracy than the one-stage method. However, it also has some drawbacks: it requires computation of a large number of redundant non-interacting human pairs, which significantly affects detection efficiency; and it relies solely on human and object features obtained from object detectors and hand-crafted spatial features for interaction recognition, failing to exploit key visual contextual cues to identify complex interactions. Summary of the Invention

[0004] The embodiment of the present application provides a method for detecting human interaction based on content query and location query, which can solve the problem that the current two-stage human interaction detection cannot fully utilize visual context clues to identify complex interaction relationships.

[0005] In the first aspect, an embodiment of the present application provides a method for detecting human interaction based on content query and position query, comprising the following steps: S1, inputting the image to be detected into the backbone network to obtain the visual features of the image, flattening it into multiple one-dimensional vectors, enhancing it through sinusoidal position coding, and passing it into the target detector DETR to obtain all human instances and object instances; S2, pairing the human instances and object instances one by one to obtain multiple human pairs, extracting the multimodal features of the human pairs, wherein the multimodal features include appearance features, spatial features and object category semantic features, connecting the appearance features, the spatial features and the object category semantic features and passing them into a multi-layer perceptron to obtain the interactivity score of each human pair, and taking the score with the highest value as the first character pair. Multiple character pairs with high probability are determined as multiple character interaction proposals; S3, the multimodal features of the character pairs extracted in S2 are deeply fused to obtain the content embedding of the character interaction proposal, and the center point coordinates of the person frame and the object frame are used to construct the position embedding of the character interaction proposal; S4, the content embedding and position embedding of multiple character interaction proposals are passed into the preset Transformer decoder to generate content queries and position queries, and decoupled cross-attention calculations are performed with image features and image position encoding to obtain interaction features; S5, the interaction features are passed into the action classifier to obtain the score of each action category, and the confidence scores of the person instance and object instance obtained in S1 are combined to obtain the score of the character interaction triplet.

[0006] In a possible implementation of the first aspect, the method of S1 is specifically:

[0007] S101, the original features of the image to be detected Input CNN backbone network to get feature map ,in is a set of real numbers, W and H are the width and height of the image to be detected, W' and H' are the width and height of the feature map, and D is the number of channels of the feature map;

[0008] S102. Calculate the two-dimensional position encoding of the feature map F:

[0009] The position code corresponding to the (x,y) position of the feature map F is , flatten the feature map F to obtain the feature sequence , let the feature sequence Add the corresponding position code to get the feature sequence with position information ;

[0010] S103, the feature sequence Input the target detector DETR to obtain all human instances and object instances in the image to be detected. Each instance is represented as , where b represents the bounding box of the instance, c represents the instance category, s represents the confidence score, and x represents the feature of the instance output by the DETR decoder.

[0011] Optionally, in another possible implementation of the first aspect, the method in S2 is specifically:

[0012] S201, pairing the detected person instances and object instances one by one to obtain multiple person pairs;

[0013] S202. For each person pair, calculate its appearance features ,in and are the features of person instances and object instances respectively;

[0014] S203. For each person pair, calculate its spatial features:

[0015] in is the center point coordinate of the human frame, is the center point coordinate of the object frame, For the width and height of the human frame, are the width and height of the object frame, is the intersection-over-union ratio of the person frame and the object frame, Represents the absolute value operation;

[0016] S204. For each person pair, calculate its object category semantic features : Construct a text prompt in the form of a photo of a / an [object], where [object] is the object category. The text prompt is passed to the text encoder of CLIP to obtain the text embedding, which is used as the semantic feature of the object category;

[0017] S205: The appearance features, spatial features, and object category semantic features of the person pairs are concatenated and passed into a multi-layer perceptron. The interactivity score is output by a Sigmoid activation function, and the person pairs with the highest scores are used as person interaction proposals.

[0018] Optionally, in another possible implementation of the first aspect, the method in S3 is specifically as follows:

[0019] S301: For each character interaction proposal, construct content embedding using the multimodal features of the character pair extracted in S2 , as follows:

[0020] in The appearance characteristics of the representative character pair, The spatial characteristics of the representative character pair, represents the object category semantic features of person pairs, and MBF stands for multi-branch fusion module;

[0021] S302: For each person interaction proposal, construct a position embedding using the center point coordinates of the person frame and the object frame. , as follows:

[0022] in is the coordinate of the center point of the human frame, is the coordinate of the center point of the object frame, The role of is to map the scalar to the sine embedding space, which is calculated as , where i =1,2,3,...,d / 2.

[0023] Optionally, in another possible implementation of the first aspect, the method in S4 is specifically as follows:

[0024] S401: Pass the content embedding and position embedding of multiple character interaction proposals into a preset Transformer decoder, which consists of 8 decoder layers;

[0025] S402: The current decoder layer receives multiple embeddings output by the previous decoder layer as the input sequence E. After the input sequence E is processed by self-attention calculation, residual connection, and layer normalization, the updated input sequence is obtained. ;

[0026] S403, input sequence As content query, the position embedding obtained by S3 is used as the position query, and multiple content queries and position queries are connected to obtain the query for cross attention calculation , that is, the i-th query The calculation method is: ,in represents the i-th content query, Represents the i-th position query; connect each feature point in the feature sequence F' obtained by S1 with the position code corresponding to the feature point to obtain the key of cross attention , that is, the i-th key The calculation method is: ,in represents the i-th feature point in the feature sequence F', Represents the position code corresponding to the i-th feature point; calculate the attention weight a of q for k by the dot multiplication method, that is, the i-th query For the jth key The attention weight The calculation method is: ; Take the feature sequence F' as the cross-attention value v, and obtain the output of the cross-attention part through weighted summation of attention weights. Then, after residual connection, layer normalization and feedforward neural network, the output embedding of the decoder layer is obtained and input to the next decoder layer. The output embedding of the last decoder layer is used as the interaction feature of the character interaction proposal.

[0027] Optionally, in another possible implementation of the first aspect, the method in S5 is specifically as follows:

[0028] S501: The interaction features of each character interaction proposal are passed to the action classifier. The action classifier is a multi-layer perceptron equipped with a Sigmoid activation function. The score of each action category is obtained. The score s of the character interaction triplet is obtained by combining the confidence scores of the person instance and object instance obtained in S1. The calculation method is as follows:

[0029] in is the confidence score of the person instance, is the confidence score of the object instance, is the score of the action category.

[0030] Beneficial effect: In the technical solution of the present application, the image to be detected is first input into the backbone network to extract the visual features of the image, and enhanced by sinusoidal position coding, and then input into the target detector DETR to obtain human and object instances, and then pair the human and object instances, extract their appearance features, spatial features and object category semantic features, and input into the multi-layer perceptron to calculate the interactivity score, from which multiple pairs of people with the highest scores are selected as interaction proposals, and then the multimodal features of the person pairs are deeply fused to generate content embedding, and position embedding is constructed based on the center coordinates of the person frame and the object frame, and then the content embedding and position embedding are input into the Transformer decoder to generate content query and position query, and cross-attention calculation is performed to obtain interaction features, and finally the interaction features are input into the action classifier, combined with the image features and confidence scores, to calculate and output the score of the person interaction triplet. Thus, by deeply fusing multimodal features and cross-attention mechanism, the detection accuracy of the interaction between people and objects is effectively improved, and the interaction relationship between people in complex scenes can be accurately identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0032] Figure 1 This is a flowchart of a method for detecting human interaction based on content query and location query provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0034] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0035] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0037] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0038] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0039] The following describes in detail the human interaction detection method based on content query and location query provided by the present application with reference to the accompanying drawings.

[0040] Figure 1 A flow chart of a method for detecting human interaction based on content query and location query provided in an embodiment of the present application is shown.

[0041] like Figure 1 As shown, the human interaction detection method based on content query and location query includes the following steps:

[0042] S1. Input the image to be detected into the backbone network to obtain the image visual features, flatten them into multiple one-dimensional vectors, enhance them through sinusoidal position encoding, and input them into the target detector DETR to obtain all human instances and object instances;

[0043] Furthermore, in the embodiment of the present application, the above method of S1 is specifically as follows:

[0044] S101, the original features of the image to be detected Input CNN backbone network to get feature map ,in is a set of real numbers, W and H are the width and height of the image to be detected, W' and H' are the width and height of the feature map, and D is the number of channels of the feature map;

[0045] S102. Calculate the two-dimensional position encoding of the feature map F:

[0046] The position code corresponding to the (x,y) position of the feature map F is , flatten the feature map F to obtain the feature sequence , let the feature sequence Add the corresponding position code to get the feature sequence with position information ;

[0047] S103, the feature sequence F'' Input the target detector DETR to obtain all human instances and object instances in the image to be detected. Each instance is represented as , where b represents the bounding box of the instance, c represents the instance category, s represents the confidence score, and x represents the feature of the instance output by the DETR decoder.

[0048] S2. Pairing person instances and object instances one by one to obtain multiple person pairs, extracting multimodal features of the person pairs, which include appearance features, spatial features, and object category semantic features. These features are concatenated and fed into a multilayer perceptron to obtain an interactivity score for each person pair. The person pairs with the highest scores are identified as multiple person interaction proposals.

[0049] Furthermore, in the embodiment of the present application, the above-mentioned method of S2 is specifically as follows:

[0050] S201, pairing the detected person instances and object instances one by one to obtain multiple person pairs;

[0051] S202. For each person pair, calculate its appearance features ,in and are the features of person instances and object instances respectively;

[0052] S203. For each person pair, calculate its spatial features:

[0053] in is the center point coordinate of the human frame, is the center point coordinate of the object frame, For the width and height of the human frame, are the width and height of the object frame, is the intersection-over-union ratio of the person frame and the object frame, Represents the absolute value operation;

[0054] S204. For each person pair, calculate its object category semantic features : Construct a text prompt in the form of a photo of a / an [object], where [object] is the object category. The text prompt is passed to the text encoder of CLIP to obtain the text embedding, which is used as the semantic feature of the object category;

[0055] S205: The appearance features, spatial features, and object category semantic features of the person pairs are concatenated and passed into a multi-layer perceptron. The interactivity score is output by a Sigmoid activation function, and the person pairs with the highest scores are used as person interaction proposals.

[0056] It should be noted that during training, the loss function of interactive prediction is , where FL represents the Focal Loss function, Indicates whether the i-th person pair is a positive sample with interaction (for a person pair, if the intersection of its person frame and object frame with the person frame and object frame in the ground truth is greater than 0.5, then the person pair is considered a positive sample, otherwise it is a negative sample). represents the predicted interactivity score of the i-th person pair.

[0057] S3, deeply fuse the multimodal features of the person pairs extracted in S2 to obtain the content embedding of the person interaction proposal, and use the center point coordinates of the person box and object box to construct the position embedding of the person interaction proposal;

[0058] Furthermore, in the embodiment of the present application, the above-mentioned method of S3 is specifically as follows:

[0059] S301: For each character interaction proposal, construct content embedding using the multimodal features of the character pair extracted in S2 , as follows:

[0060] in The appearance characteristics of the representative character pair, The spatial characteristics of the representative character pair, represents the object category semantic features of person pairs, and MBF stands for multi-branch fusion module;

[0061] S302: For each person interaction proposal, construct a position embedding using the center point coordinates of the person frame and the object frame. , as follows:

[0062] in is the coordinate of the center point of the human frame, is the coordinate of the center point of the object frame, The role of is to map the scalar to the sine embedding space, which is calculated as , where i =1,2,3,...,d / 2.

[0063] S4. Pass the content embeddings and position embeddings of multiple character interaction proposals into the preset Transformer decoder to generate content queries and position queries. These queries are then decoupled with image features and image position encodings for cross-attention calculation to obtain interaction features.

[0064] Furthermore, in the embodiment of the present application, the method of S4 is specifically as follows:

[0065] S401: Pass the content embedding and position embedding of multiple character interaction proposals into a preset Transformer decoder, which consists of 8 decoder layers;

[0066] S402: The current decoder layer receives multiple embeddings output by the previous decoder layer as the input sequence E. After the input sequence E is processed by self-attention calculation, residual connection, and layer normalization, the updated input sequence is obtained. ;

[0067] S403, input sequence As content query, the position embedding obtained by S3 is used as the position query, and multiple content queries and position queries are connected to obtain the query for cross attention calculation , that is, the i-th query The calculation method is: ,in represents the i-th content query, Represents the i-th position query; connect each feature point in the feature sequence F' obtained by S1 with the position code corresponding to the feature point to obtain the key of cross attention , that is, the i-th key The calculation method is: ,in represents the i-th feature point in the feature sequence F', Represents the position code corresponding to the i-th feature point; calculate the attention weight a of q for k by the dot multiplication method, that is, the i-th query For the jth key The attention weight The calculation method is: ; Take the feature sequence F' as the cross-attention value v, and obtain the output of the cross-attention part through weighted summation of attention weights. Then, after residual connection, layer normalization and feedforward neural network, the output embedding of the decoder layer is obtained and input to the next decoder layer. The output embedding of the last decoder layer is used as the interaction feature of the character interaction proposal.

[0068] It should be noted that from the i-th query For the jth key The attention weight Calculation method: It can be seen that by decoupling the query into content query and location query, the attention mechanism can be guided from different aspects: content query can obtain relevant information from the image based on the person features, while location query can explicitly guide the model to focus on image features near the center points of the person box and object box, which is more conducive to discovering visual context clues.

[0069] S5: Pass the interaction features into the action classifier to obtain the score of each action category. Combined with the confidence scores of the person instance and object instance obtained in S1, the score of the person interaction triplet is obtained.

[0070] Furthermore, in the embodiment of the present application, the method of S5 is specifically as follows:

[0071] S501: The interaction features of each character interaction proposal are passed to the action classifier. The action classifier is a multi-layer perceptron equipped with a Sigmoid activation function. The score of each action category is obtained. The score s of the character interaction triplet is obtained by combining the confidence scores of the person instance and object instance obtained in S1. The calculation method is as follows:

[0072] in is the confidence score of the person instance, is the confidence score of the object instance, is the score of the action category.

[0073] As a possible implementation method, during training, the total loss function of the entire network is is calculated as follows:

[0074]

[0075] Where C represents the number of interaction categories, N represents the number of character interaction proposals, Indicates whether the label corresponding to the interaction proposal of the i-th character exists in the c-th interaction category, The probability that the cth interaction category exists for the predicted i-th character interaction proposal, is the loss of interactivity prediction in S2.

[0076] The human interaction detection method based on content query and position query provided by this application first inputs the image to be detected into the backbone network, extracts the image visual features, and enhances them through sinusoidal position encoding, and then inputs them into the target detector DETR to obtain human and object instances. Then, the human and object instances are paired, their appearance features, spatial features and object category semantic features are extracted, and they are input into the multi-layer perceptron to calculate the interactivity score. Multiple human pairs with the highest scores are selected as interaction proposals. Then, the multimodal features of the human pairs are deeply fused to generate content embedding, and position embedding is constructed based on the center coordinates of the human frame and the object frame. The content embedding and position embedding are then input into the Transformer decoder to generate content query and position query, and cross-attention calculation is performed to obtain interaction features. Finally, the interaction features are input into the action classifier, and the image features and confidence scores are combined to calculate and output the score of the human interaction triplet. Thus, by deeply fusing multimodal features and cross-attention mechanism, the detection accuracy of human-object interaction is effectively improved, and the human interaction relationship in complex scenes can be accurately identified.

[0077] Part of the core code of this application's human interaction detection method based on content query and location query is as follows:

[0078] (1) Calculate the spatial features of person pairs:

[0079] def compute_spatial_features(

[0080] boxes_1: List[Tensor], boxes_2: List[Tensor], eps: float = 1e-10

[0081] ) ->Tensor:

[0082] features = []

[0083] for b1, b2, shape in zip(boxes_1, boxes_2):

[0084] c1_x = (b1[:, 0] + b1[:, 2]) / 2; c1_y = (b1[:, 1] + b1[:, 3]) / 2

[0085] c2_x = (b2[:, 0] + b2[:, 2]) / 2; c2_y = (b2[:, 1] + b2[:, 3]) / 2

[0086] b1_w = b1[:, 2] - b1[:, 0]; b1_h = b1[:, 3] - b1[:, 1]

[0087] b2_w = b2[:, 2] - b2[:, 0]; b2_h = b2[:, 3] - b2[:, 1]

[0088] d_x = torch.abs(c2_x - c1_x) / (b1_w + eps)

[0089] d_y = torch.abs(c2_y - c1_y) / (b1_h + eps)

[0090] iou = torch.diag(box_ops.box_iou(b1, b2))

[0091] f = torch.stack([

[0092] d_x, d_y,

[0093] b2_w * b2_h / (b1_w * b1_h + eps),

[0094] b1_w / (b1_h + eps), b2_w / (b2_h + eps),

[0095] iou

[0096] ], 1)

[0097] features.append(f)

[0098] return torch.cat(features)

[0099] (2) Calculate the CLIP text embedding corresponding to each object category:

[0100] with torch.no_grad():

[0101] device = "cuda" if torch.cuda.is_available() else "cpu"

[0102] clip_model = 'ViT-B / 32'

[0103] self.clip_model, preprocess = clip.load(clip_model, device=device)

[0104] obj_text_inputs = torch.cat([clip.tokenize(obj_text[1]) for obj_text in obj_text_label])

[0105] self.obj_text_embedding = self.clip_model.encode_text(obj_text_inputs.to(device)).float()

[0106] del clip_model

[0107] (3) Calculate the content embedding of the character interaction proposal:

[0108] # mbf and mbf_as are two different multi-branch fusion modules

[0109] content_embed = self.mbf(

[0110] sem_features[x_keep, y_keep],

[0111] self.mbf_as(

[0112] torch.cat([embeds[x_keep], embeds[y_keep]], dim=1),

[0113] spatial_features[x_keep, y_keep] ) )

[0116] (4) Calculate the position embedding of the character interaction proposal:

[0117] def compute_c_pe(boxes, image_size):

[0118] bx_norm = boxes / image_size[[1, 0, 1, 0]]

[0119] bx_c = (bx_norm[:, :2] + bx_norm[:, 2:]) / 2

[0120] c_pe = compute_sinusoidal_pe(bx_c[:, None], 10000).squeeze(1)

[0121] return c_pe

[0122] c_pe = compute_c_pe(boxes, image_size)

[0123] position_embed = torch.add(c_pe[x_keep], c_pe[y_keep]).unsqueeze(1)

[0124] (5) Cross-attention calculation for decoupling content query and location query:

[0125] # tgt is the embedding output of the self-attention part

[0126] q_content = self.ca_qcontent_proj(tgt)

[0127] # memory is image features

[0128] k_content = self.ca_kcontent_proj(memory)

[0129] v = self.ca_v_proj(memory)

[0130] num_queries, bs, n_model = q_content.shape

[0131] hw, _, _ = k_content.shape

[0132] #pos is the image position code

[0133] k_pos = self.ca_kpos_proj(pos)

[0134] q = q_content

[0135] k = k_content

[0136] q = q.view(num_queries, bs, self.nhead, n_model / / self.nhead)

[0137] # query_sine_embed is the position embedding proposed for character interaction

[0138] query_sine_embed = self.ca_qpos_sine_proj(query_sine_embed)

[0139] query_sine_embed = query_sine_embed.view(num_queries, bs, self.nhead,n_model / / self.nhead)

[0140] q = torch.cat([q, query_sine_embed], dim=3).view(num_queries, bs, n_model * 2)

[0141] k = k.view(hw, bs, self.nhead, n_model / / self.nhead)

[0142] k_pos = k_pos.view(hw, bs, self.nhead, n_model / / self.nhead)

[0143] k = torch.cat([k, k_pos], dim=3).view(hw, bs, n_model * 2)

[0144] tgt2 = self.cross_attn(query=q, key=k, value=v, attn_mask=memory_mask, key_padding_mask=memory_key_padding_mask)[0]

[0145] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0146] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for detecting human interaction based on content query and location query, characterized in that: The steps include: S1. Input the image to be detected into the backbone network to obtain the image visual features, flatten them into multiple one-dimensional vectors, enhance them through sinusoidal position encoding, and input them into the target detector DETR to obtain all human instances and object instances; S2. Pairing person instances and object instances one by one to obtain multiple person pairs, extracting multimodal features of the person pairs, wherein the multimodal features include appearance features, spatial features, and object category semantic features, concatenating the appearance features, spatial features, and object category semantic features and inputting the features into a multilayer perceptron to obtain an interactivity score for each person pair, and determining the multiple person pairs with the highest scores as multiple person interaction proposals; S3, deeply fuse the multimodal features of the person pairs extracted in S2 to obtain the content embedding of the person interaction proposal, and use the center point coordinates of the person box and object box to construct the position embedding of the person interaction proposal; S4. Pass the content embeddings and position embeddings of multiple character interaction proposals into the preset Transformer decoder to generate content queries and position queries. These queries are then decoupled with image features and image position encodings for cross-attention calculation to obtain interaction features. S5: Pass the interaction features into the action classifier to obtain the score of each action category. Combined with the confidence scores of the person instance and object instance obtained in S1, the score of the person interaction triplet is obtained. The S3 method is as follows: S301, for each character interaction proposal, use the multimodal features of the character pair extracted in S2 to build the content embedding e c , as follows: f as =MBF(f a ,f s ) f pair =MBF(f as ,f c ) e c =f pair where f a The appearance characteristics of the representative character pair, f s represents the spatial characteristics of the person pair, f c represents the object category semantic features of person pairs, and MBF stands for multi-branch fusion module; S302: For each character interaction proposal, construct a position embedding using the center point coordinates of the person frame and the object frame. p , as follows: e p =[φ(x h )+φ(x o );φ(y h )+φ(y o )] where x h ,y h is the coordinate of the center point of the human frame, x o ,y o is the coordinate of the center point of the object frame, and the function of φ(·) is to map the scalar to the sinusoidal embedding space, which is calculated as follows: Where i = 1, 2, 3, ..., d / 2.

2. The method according to claim 1, characterized in that The method of S1 is specifically as follows: S101, the original features of the image to be detected Input CNN backbone network to get feature map in is a set of real numbers, W and H are the width and height of the image to be detected, W' and H' are the width and height of the feature map, and D is the number of channels of the feature map; S102. Calculate the two-dimensional position encoding of the feature map F: The position code corresponding to the (x, y) position of the feature map F is [pe x ;pe y ], flatten the feature map F to obtain the feature sequence Let the feature sequence F' add the corresponding position code to obtain the feature sequence with position information S103. Input the feature sequence F" into the target detector DETR to obtain all human instances and object instances in the image to be detected. Each instance is represented as (b, c, s, x), where b represents the bounding box of the instance, c represents the instance category, s represents the confidence score, and x represents the feature of the instance output by the DETR decoder.

3. The method according to claim 1, characterized in that The method of S2 is specifically as follows: S201, pairing the detected person instances and object instances one by one to obtain multiple person pairs; S202. For each person pair, calculate its appearance feature f a =[x h ;x o ], where x h and x o are the features of person instances and object instances respectively; S203. For each person pair, calculate its spatial features: in is the center point coordinate of the human frame, is the center point coordinate of the object frame, W h ,H h is the width and height of the human frame, W o ,H o is the width and height of the object box, iou is the intersection-union ratio of the human box and the object box, and abs(·) represents the absolute value operation; S204. For each person pair, calculate its object category semantic feature f c : Construct a text prompt in the form of a photo of a / an [object], where [object] is the object category. The text prompt is passed to the text encoder of CLIP to obtain the text embedding, which is used as the semantic feature of the object category; S205: The appearance features, spatial features, and object category semantic features of the person pairs are concatenated and passed into a multi-layer perceptron. The interactivity score is output by a Sigmoid activation function, and the person pairs with the highest scores are used as person interaction proposals.

4. The method according to claim 2, characterized in that The method of S4 is as follows: S401: Transmit the content embedding and position embedding of multiple character interaction proposals into a preset Transformer decoder, where the preset Transformer decoder consists of 8 decoder layers; S402: The current decoder layer receives multiple embeddings output by the previous decoder layer as an input sequence E, and performs self-attention calculation, residual connection, and layer normalization on the input sequence E to obtain an updated input sequence E'; S403: Take the input sequence E' as the content query, take the position embedding obtained in S3 as the position query, concatenate multiple content queries and position queries respectively, and obtain the query q = (q1, q2, ..., q k ), that is, the i-th query q i The calculation method is: in represents the i-th content query, Represents the i-th position query; connect each feature point in the feature sequence F' obtained by S1 with the position code corresponding to the feature point to obtain the cross attention key k=(k1,k2,…,k n ), that is, the i-th key k i The calculation method is: in represents the i-th feature point in the feature sequence F', Represents the position code corresponding to the i-th feature point; calculate the attention weight a of q for k by the dot multiplication method, that is, the i-th query q i For the jth key k j The attention weight a ij The calculation method is: The feature sequence F' is used as the cross-attention value v, and the output of the cross-attention part is obtained by weighted summation of the attention weights. Then, after residual connection, layer normalization and feedforward neural network, the output embedding of the decoder layer is obtained and input into the next decoder layer. The output embedding of the last decoder layer is used as the interaction feature of the character interaction proposal.

5. The method according to claim 1, wherein The method of S5 is as follows: S501: The interaction features of each character interaction proposal are passed to the action classifier, which is a multi-layer perceptron equipped with a Sigmoid activation function. The score of each action category is obtained, and the confidence scores of the person instance and object instance obtained in S1 are combined to obtain the score s of the character interaction triplet. The calculation method is as follows: s=s h s o s a where s h The confidence score of the person instance, s o is the confidence score of the object instance, s a is the score of the action category.

Images