An efficient self-adaptive picture human interaction detection method and system

Abstract

The present application relates to a kind of high-efficiency adaptive picture character interaction detection method and system.The system contains two kinds of operating modes: training-free mode: system utilizes pre-trained target detector and multimodal model, constructs the concept guide of multi-branch character interaction memory module, to simultaneously store specific domain visual knowledge and universal domain semantic knowledge, complete character interaction detection;Fine-tuning mode: utilize lightweight adapter based on attention mechanism, for the visual encoder in model is incorporated into instance-level priori knowledge, while updating the knowledge stored in character interaction memory module, obtain better character coding features.The model under training-free in the present application can complete character interaction detection task and obtain competitive performance in the case of only a small amount of samples, the model under fine-tuning mode can obtain more spatially sensitive pair relationship features, so that character interaction relationship detection is more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of picture detection, and in particular to an efficient adaptive picture character interaction detection method and system based on concept-guided memory. BACKGROUND

[0002] Character interaction detection is an important and challenging task, which has potential application value in the fields of video surveillance, robots, etc. Picture character interaction detection technology is a technology that analyzes and understands the content of a picture, locates all people and objects in the picture, and determines the interaction relationship between them. This technology mainly includes one-stage and two-stage methods.

[0003] One-stage methods directly detect <person, object, interaction relationship> triplets in the picture through an end-to-end model. Representative methods include: UnionNet algorithm proposed by Kim, B. et al., which directly focuses on the features of the union region and eliminates additional inference; AS-Net (Adaptive Set-based one-stage framework) algorithm proposed by Mingfei Chen et al., which maps interaction queries to interaction predictions and adaptively matches labels; HOTR (Human-Object interaction TRansformer) algorithm proposed by Bumsoo Kim et al., which performs character detection and relationship prediction in parallel; QPIC (Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information) algorithm proposed by Masato Tamura et al., which is based on queries and avoids the mutual influence of different characters; GEN-VLKT (Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection) algorithm proposed by Yue Liao et al., which independently represents the queries of people and objects and uses a multi-modal pre-training model. However, these methods require the model to learn a same feature representation for both character instance detection and character relationship prediction, thus being limited by the multi-task learning ability.

[0004] The two-stage method first detects all people and objects in the picture, and then predicts the interaction relationship between them. Representative methods include: TIN (Transferable Interactiveness Network) algorithm for binary interaction prediction proposed by Yong-Lu Li et al., IDN (Integration-Decomposition Network) algorithm that can recombine human and object features proposed by Yong-Lu Li et al., STIP (Structure-aware Transformer over Interaction Proposals) algorithm that focuses on spatial human structure information proposed by Yong Zhang et al., and UPT (Unary-Pairwise Transformer) algorithm that unifies unary and pairwise representations proposed by Frederic Z Zhang et al. The two-stage method decouples the tasks of person detection and relationship classification, allowing each stage to focus on its own task and produce good results in each stage.

[0005] However, the above one-stage and two-stage methods require a large amount of balanced training data, and the training process often consumes a lot of computing resources.

[0006] Visual-language pre-training is a very basic work in the field of artificial intelligence. In recent years, thanks to the introduction of attention mechanisms, the field has made rapid progress. Among them, CLIP (Contrastive Language-Image Pre-Training) is a neural network model trained on a large number of <image, text> pairs, which has strong zero-shot learning ability in picture classification and other tasks, and has been proven effective in many downstream tasks. For example, Clip-adapter algorithm proposed by Peng Gao et al. only needs to add an adapter to the last few layers of the model, Adaptformer algorithm proposed by Shoufa Chen et al. inserts a learnable subnetwork in each module of the model. However, these methods do not allow the model to focus on paired spatial relationships, so they are not suitable for the task of human interaction detection. SUMMARY

[0007] In view of the shortcomings of the prior art, the present application proposes an efficient adaptive human interaction detection method and system based on concept-guided memory.

[0008] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0009] An efficient adaptive picture human interaction detection method based on concept-guided memory, which is used to detect all people and objects in a picture and predict their interaction relationship, includes the following two operation modes:

[0010] (1) Training-free mode: a multi-branch concept-guided human interaction memory module is constructed by using a pre-trained object detector and a multi-modal model to simultaneously store domain-specific visual knowledge and domain-agnostic semantic knowledge based on concept-guided human interaction, so that the model can complete the human interaction detection task and achieve competitive performance even with a small number of samples.

[0011] (2) Fine-tuning mode: a lightweight adapter based on attention mechanism is used to integrate instance-level prior knowledge into the visual encoder in the model, and the knowledge stored in the human interaction memory module is updated to obtain more spatially sensitive pair relationship features, so that the human interaction relationship prediction is more accurate.

[0012] Further, step (1) uses a pre-trained object detector DETR to detect all people and objects in the picture, and a multi-modal pre-training model CLIP (Contrastive Language-Image Pretraining) to extract features for the detected target regions and language descriptions of different human interaction categories.

[0013] Further, the domain-specific visual knowledge of step (1) requires three types of target region features, namely the region of a person, the region of an object, and the union region composed of the union of the above two regions.

[0014] Further, the human interaction memory module in step (1) includes three branches, namely the instance-centric branch, the interaction-aware branch, and the semantic branch. The instance-centric branch stores a set of fine-grained features centered on people and objects, i.e., the features of the person region and the object region; the interaction-aware branch stores features containing background information of human interaction scenes, i.e., the features of the union region of people and objects, and the two branches store features that constitute domain-specific visual knowledge. In addition, the semantic branch stores a set of semantic features of various human interactions with generalization ability, i.e., domain-agnostic semantic knowledge.

[0015] Further, the instance center branch and the interaction awareness branch of step (1) are composed of a dictionary-like <key, value> structure, the corresponding extracted features as keys, and the corresponding labels as values; and the semantic branch is a classifier initialized by the semantic encoding of each person-object interaction as a weight.

[0016] Further, step (2) is based on the initialization of the concept-guided person-object interaction memory module constructed in step (1), so that the keys of the instance center branch and the interaction awareness branch, and the weights of the semantic branch classifier become three learnable parameters, while the values of the instance center branch and the interaction awareness branch remain fixed, so that the model can select the memory content that is emphasized or suppressed, i.e. the three learnable parameters.

[0017] Further, step (2) introduces a lightweight prior knowledge injection adapter (Instance-aware Adapter) to integrate instance-level prior knowledge into the spatial feature map of the early visual encoder, so that the model can pay more attention to the spatial position features of people and objects and better serve the pair relationship prediction task.

[0018] Further, the instance-level prior knowledge of step (2) includes semantic features, i.e. semantic encoding of instances, which provides language prior knowledge for the visual encoder to prompt it about which interaction types the current object may contain; spatial position features, which provide geometric spatial position information of instances in the picture, allowing the visual encoder to distinguish different instance types in a fine-grained manner; and confidence scores, which reflect the uncertainty of the current instance judgment.

[0019] Further, in order to make the model lightweight, step (2) first maps the dimensions and feature representations of prior knowledge to the same shared low-dimensional space through low-dimensional mapping, and then fuses the prior knowledge into the visual encoder through attention mechanism to obtain better person visual encoding features.

[0020] Further, step (2) compares the cosine similarity of the person visual encoding features with the keys of the instance center branch and the interaction awareness branch in the person-object interaction memory module, and respectively weights the values of each branch to obtain the prediction results of the above two branches, and passes the person visual encoding features through the semantic branch classifier to obtain the prediction result of this branch. Finally, the prediction results of different branches are fused in a weighted average manner to obtain the final interaction category prediction result, and the model uses focal loss to alleviate the imbalance problem between positive and negative samples.

[0021] An efficient and adaptive picture person-object interaction detection system based on concept-guided memory using the above method, comprising:

[0022] The untrained module utilizes a pre-trained target detector and a multi-modal model to construct a multi-branch, concept-guided human interaction memory module to simultaneously store specific domain visual knowledge and universal domain semantic knowledge, so that the model can also complete the task interaction detection task in the case of only a small number of samples.

[0023] The fine-tuning module utilizes a lightweight adapter based on an attention mechanism to integrate instance-level prior knowledge into the visual encoder in the model, and simultaneously updates the knowledge stored in the human interaction memory module, to obtain more spatially sensitive paired human features, and improve the human interaction detection accuracy by learning only a small amount of parameters.

[0024] The beneficial effects of the present application are:

[0025] On the one hand, the present application can make full use of existing pre-trained models, so that the model can simultaneously utilize specific domain visual knowledge and universal domain semantic knowledge based on human interaction concept guidance, and complete the human interaction detection task without training and only using a small number of labeled samples; on the other hand, in order to efficiently adapt to a new domain, the present application proposes to integrate prior knowledge into the model, so that the model can be efficiently transferred to a new domain through lightweight training. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flowchart of the efficient adaptive picture human interaction detection method of the present application.

[0027] Figure 2 is a lightweight adapter structure diagram that can integrate instance-level prior information. DETAILED DESCRIPTION

[0028] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0029] A kind of efficient adaptive picture human interaction detection method of the present application, its flow as shown in Figure 1 Specifically includes the following modes:

[0030] (1) Untrained mode

[0031] In the untrained mode, the model utilizes a pre-trained target detector DETR to detect all people and objects in the picture, and utilizes a pre-trained CLIP to extract features for the target region and the language description of different human interaction categories. The region of the person and the region of the object detected by the target detector can be denoted as b h ,b o The union region of each pair of people and objects can be denoted as b uThe features extracted by CLIP for the region of the person, the region of the object, and the region of the person-object set can be denoted as f h ,f o ,f u The invention combines f h and f o to form the object-centered fine-grained feature f IC , and directly uses f u as the feature f IA containing the background information of the person-object interaction scene.

[0032] For the instance-centered branch <F IC , L IC >, where F IC and L IC represent the key and value in the dictionary structure in this branch, the invention stores the instance-centered feature f IC extracted from the training set image into F IC , and converts the corresponding label into a multi-hot vector and stores it into L IC .

[0033] For the interaction-aware branch <F IA , L IA >, similarly, F IA and L IA represent the key and value in the dictionary structure in this branch, the invention stores the feature f u rich in interaction information and the spatial position relationship of the person-object into F IA , and converts the corresponding label into a multi-hot vector and stores it into L IA .

[0034] For the semantic branch, the invention aims to use language common sense to improve the performance of interaction category prediction. The classifier of this branch can be denoted as W T . To construct W T , first, a prompt sentence is manually designed for each person-object interaction category, such as “a picture of a person holding an object”, the prompt sentence is sent into the CLIP language encoder to obtain its encoding vector in the visual-linguistic common feature space, which can be used as the representative vector of this action category and stored into W T as the classifier weight. The semantic branch can be easily generalized to new person-object interaction categories by adding the semantic encoding of new categories through the use of visual-linguistic common feature space representation.

[0035] In the test stage, for each person-object pair in the test image, the features of each region are extracted to form f IC ,f IAand further obtain the interaction class prediction of the person-object pair through the following formula:

[0036]

[0037]

[0038] wherein γ IC , γ IA , γ T are weight coefficients of the three branches respectively, is the interaction class prediction corresponding to the current person and object.

[0039] (2) Fine-tuning mode

[0040] In the fine-tuning mode, the priori knowledge is fused into the visual encoder through a lightweight adapter. The structure of the lightweight adapter is shown in Figure 2 .

[0041] The priori knowledge includes three parts, which are: semantic features, i.e. semantic encoding of instances, providing language priori knowledge for the visual encoder, which can prompt it to know which interaction types the current object may contain; spatial position features, providing geometric spatial position information of instances in the picture, so that the visual encoder can distinguish different instance types in a fine-grained manner; and confidence score, reflecting the uncertainty of the current instance judgment.

[0042] The instance-level adapter is a lightweight sub-module that can be inserted into each layer component of the visual encoder to help the model learn to fuse priori knowledge from the objects detected by the target detector, so as to make the visual encoder better serve the downstream tasks.

[0043] Further, let represent the i-th layer feature map of the visual encoder, wherein H'W' represents the size of the feature map, and d represents the spatial dimension of the feature map. In order to reduce the parameter amount and calculation cost, a low-dimensional mapping matrix and a high-dimensional mapping matrix are used to realize the compression of the feature matrix to a low-dimensional space or the mapping back to a high-dimensional space, wherein d' << d. Then, a multi-head cross attention mechanism (Multi-head cross attention, abbreviated as MHSA) is used to realize the priori knowledge injection:

[0044] H t =MLP p (P t ) (3)

[0045] X′ i =MHSA((X i ·W down ),Ht ,H t )·W up (4)

[0046] where X' i represents prior knowledge, represents N t instance-level prior knowledge extracted from the picture, p t ={b t ,c t ,e t}, composed of the bounding box coordinates b t of the instance, the confidence c t of the instance and the semantic encoding e t of the instance. H t represents the hidden state after the prior knowledge is mapped through the fully connected network MLP p , and the reduced dimension feature map of the visual encoder is in the same feature space, where H t is the Key and Value in the multi-head attention mechanism, and the reduced dimension feature map X i ·W down is the Query in the multi-head attention mechanism.

[0047] As shown in Fig. 1, during the training process, the weights of the dark gray part of the model are fixed, and the weights of the light gray part are updated. The model maximizes the prediction accuracy through the focal loss Figure 1 , and updates the weights as follows:

[0048]

[0049]

[0050] where I is the given input picture, is the target detector, f is the visual encoder, ε is the visual feature of the picture after encoding, is the human interaction memory module, g is the human interaction class prediction function defined in formulas (1) and (2), is the real interaction label, θ is the parameter to be optimized of the model, and θ * is the parameter obtained after final optimization.

[0051] The following experimental results show that, compared with existing methods, the task interaction detection method of the present application can achieve higher accuracy with less training time.

[0052] ​The HICO-DET dataset is used in the embodiment. The dataset is composed of 47776 images, of which the training set contains 38118 images and the test set contains 9658 images. The 600 person-object interaction categories of the dataset are composed of 117 action categories and 80 object categories, involve a large number of interaction categories, and the same person can have multiple interaction relationships, which is challenging and helps to prove the effectiveness of the method.

[0053] Existing method 1: The method proposed by Bumsoo Kim et al. in the literature “Hotr: End-to-end human-object interaction detection with transformers.”

[0054] Existing method 2: The method proposed by Frederic Z Zhang et al. in the literature “Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.”

[0055] Existing method 3: The method proposed by Yue Liao et al. in the literature “Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection.”

[0056] The present application: The method of the embodiment.

[0057] The mAP index is used in the experiment to evaluate the accuracy of person interaction detection. The index measures the positioning accuracy of people and objects and the relationship prediction accuracy. A group of person interactions is considered to be successfully detected only when the positioning is accurate, the intersection over union of the detected bounding box and the real bounding box is greater than 0.5, and the interaction relationship is correctly predicted.

[0058] As can be seen from Table 1, the present application achieves better person interaction detection results with the least amount of learnable parameters. Although some existing methods also use the ability of pre-trained models, they do not establish an efficient person interaction detection system, and the model still needs to update a large number of learnable parameters.

[0059] Table 1. Comparison of experimental results with existing methods

[0060]

[0061]

[0062] Based on the same inventive concept, another embodiment of the present application provides an efficient picture human interaction detection system, which comprises:

[0063] A training-free module is configured to construct a multi-branch concept-guided human interaction memory module by using a pre-trained object detector and a multi-modal model, so that the human interaction memory module can store specific domain visual knowledge and universal domain semantic knowledge at the same time, and the human interaction detection task can be completed and competitive performance can be achieved even with a small amount of samples.

[0064] A fine-tuning module is configured to use a lightweight adapter based on an attention mechanism to integrate instance-level prior knowledge into a visual encoder, and update the knowledge stored in the human interaction memory module, so as to obtain more spatially sensitive pair relationship features, and make the human interaction relationship prediction more accurate.

[0065] Based on the same inventive concept, another embodiment of the present application provides an electronic device (computer, server, smart phone, etc.), which comprises a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method of the present application.

[0066] Based on the same inventive concept, another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program, and the computer program is executed by a computer to realize each step of the method of the present application.

[0067] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. An efficient adaptive picture person interaction detection method based on concept-guided memory, characterized in that, The method comprises the following modes: A pre-trained target detector and a multi-modal model are used to construct a multi-branch concept-guided human interaction memory module, which simultaneously stores specific domain visual knowledge and universal domain semantic knowledge, so that the human interaction detection task can be completed and competitive performance can be achieved even with only a small number of samples in a no-training mode; A lightweight adapter based on an attention mechanism is used to integrate instance-level prior knowledge into a visual encoder, and the stored knowledge in the human interaction memory module is updated to obtain more spatially sensitive pair relationship features, so that the human interaction relationship prediction is more accurate in a fine-tuning mode; The no-training mode uses a pre-trained target detector DETR to detect all people and objects in a picture, and uses a pre-trained CLIP to extract features for the target area and the language description of different human interaction categories; The regions of people and objects detected by the target detector are respectively denoted as b h , o For each pair of people and objects, the union region is denoted as b u ; the regions of people, objects, and the union of people are respectively denoted as f h , o , u , f h and f o are concatenated to form the object-centered fine-grained feature f IC ; f u is directly taken as the feature f IA containing the information of the human-object interaction scene. The human interaction memory module comprises three branches, namely an instance center branch, an interaction awareness branch and a semantic branch; the instance center branch stores a set of fine-grained features centered on people and objects, namely the features of the people area and the object area; the interaction awareness branch stores features containing background information of the human interaction scene, namely the features of the people and object union area; the features stored in the instance center branch and the interaction awareness branch constitute specific domain visual knowledge; and the semantic branch stores a set of semantic features of various human interactions with generalization capability, namely universal domain semantic knowledge; The lightweight adapter is inserted into each layer component of the visual encoder, helping to learn the fusion priori knowledge from the objects detected by the target detector; set represents the i-th layer feature map of the visual encoder, where H'W' represents the size of the feature map, d represents the spatial dimension of the feature map, and a low-dimensional mapping matrix and a high-dimensional mapping matrix compress the feature matrix into a low-dimensional space or map it back to a high-dimensional space, where d' << d; then the priori knowledge is injected by using a multi-head attention mechanism MHSA: H t = MLP p (P t ) X' i = MHSA((X i · W down ), H t , H t )· W up Where, X′ i Represents prior knowledge, N represents the number of images extracted from the image. t Prior knowledge at the instance level; p t ={b t ,c t ,e t }The bounding box coordinates b of the instance t The confidence level c of the instance t semantic encoding of instances e t Composition, H t Prior knowledge is represented by a fully connected MLP network. p The hidden state after mapping; In the fine-tuning mode, the key comparison cosine similarity between the human visual encoding features and the instance center branch and the interaction awareness branch in the human interaction memory module is calculated, and the values of the two branches are weighted by the cosine similarity to obtain the prediction results of the two branches, the human visual encoding features are input into the classifier of the semantic branch to obtain the prediction result of the semantic branch, and finally the prediction results of different branches are fused in a weighted average manner to obtain the final human interaction category prediction result; The prior knowledge is integrated into the visual encoder through an attention mechanism to obtain the human visual encoding features.

2. The method of claim 1, wherein, For the instance center branch <F IC ,L IC >, where F IC and L IC respectively represent the key and value in the dictionary structure in the branch, the instance center feature f IC is stored into F IC , and the corresponding label is converted into a multi-hot vector and stored into L IC ; for the interaction awareness branch <F IA ,L IA >, the feature f u rich in interaction information and the spatial position relationship between characters is stored into F IA , and the corresponding label is stored into L IA ; For the semantic branch, for each human interaction class, a prompt sentence is designed manually, encoded by CLIP, and stored as the classifier weights W T .

3. The method of claim 1, wherein, The instance-level prior knowledge comprises semantic features, spatial position features and confidence scores; the semantic features are semantic encodings of instances, which provide language prior knowledge for the visual encoder and prompt which interaction types the current object may contain; the spatial position features provide geometric spatial position information of the instances in the picture, so that the visual encoder can distinguish different instance types in a fine-grained manner; and the confidence scores reflect the uncertainty of the current instance judgment.

4. The method of claim 1, wherein, maximizing prediction accuracy by focal loss and updating the weights: wherein for a given input picture, is the target detector, f is the visual encoder, and ε is the encoded visual features of the picture, is the human interaction memory module, g is the human interaction class prediction function, is the true interaction label, θ is the parameter to be optimized in the model, and θ * is the final optimized parameter.

5. A high efficient adaptive picture person interactive detection system based on concept guided memory using the method of any one of claims 1-4. The method comprises: A no-training module is configured to use a pre-trained target detector and a multi-modal model to construct a multi-branch concept-guided human interaction memory module, which simultaneously stores specific domain visual knowledge and universal domain semantic knowledge, so that the human interaction detection task can be completed and competitive performance can be achieved even with only a small number of samples; A fine-tuning module is configured to use a lightweight adapter based on an attention mechanism to integrate instance-level prior knowledge into a visual encoder, and update the stored knowledge in the human interaction memory module to obtain more spatially sensitive pair relationship features, so that the human interaction relationship prediction is more accurate.

6. A computer device, comprising: comprising a memory storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the method of any one of claims 1-4.

Images