Efficient self-adaptive picture character interaction detection method and system

Abstract

The invention relates to a high-efficiency self-adaptive picture character interaction detection method and a high-efficiency self-adaptive picture character interaction detection system. The system comprises two operation modes: a training-free mode: the system uses a pre-trained target detector and a multi-modal model to construct a multi-branch concept-guided character interaction memory module to store specific domain visual knowledge and universal domain semantic knowledge at the same time to complete character interaction detection; in the fine tuning mode, a lightweight adapter based on an attention mechanism is utilized to integrate instance-level priori knowledge for a visual encoder in the model, and meanwhile, knowledge stored in a character interaction memory module is updated to obtain better character encoding features. According to the method, the model without training can complete a character interaction detection task and obtain competitive performance under the condition that only a small number of samples exist, the model in a fine tuning mode can obtain pairwise relation characteristics which are more spatially sensitive, and character interaction relation detection is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of picture detection, and in particular to a highly efficient and adaptive picture-person interaction detection method and system based on concept-guided memory. Background Art

[0002] Human interaction detection is an important and challenging task, with potential application value in video surveillance, robotics and other fields. Image human interaction detection technology is a technology that analyzes and understands the content of an image, locates all people and objects in the image, and determines the interaction relationship between them. This technology is mainly divided into two types of methods: one-stage and two-stage methods.

[0003] The one-stage method directly detects the <person, object, interaction relationship> triples in the image through an end-to-end model. Representative methods include: the UnionNet algorithm proposed by Kim, B. et al., which directly focuses on the union regional features and eliminates additional inferences; the AS-Net (Adaptive Set-based one-stage framework) algorithm proposed by Mingfei Chen et al., which maps interaction queries to interaction predictions and adaptively matches labels; the HOTR (Human-Object Interaction TRansformer) algorithm proposed by Bumsoo Kim et al., which performs person detection and relationship prediction in parallel; the QPIC (Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information) algorithm proposed by Masato Tamura et al., which avoids the influence of different pairs of people based on queries; and the GEN-VLKT (Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection) algorithm proposed by Yue Liao et al., which independently represents person queries and object queries and uses multimodal pre-trained models. However, these methods require the model to learn the same feature representation for two different tasks: person instance detection and person relationship prediction, and are therefore limited in their multi-task learning capabilities.

[0004] The two-stage method first detects all people and objects in the image, and then predicts their interactive relationships. Representative methods include: the TIN (Transferable Interactiveness Network) algorithm proposed by Yong-Lu Li et al. for interactive binary prediction, the IDN (Integration-Decomposition Network) algorithm proposed by Yong-Lu Li et al. for recombining the features of people and objects, the STIP (Structure-aware Transformer over Interaction Proposals) algorithm proposed by Yong Zhang et al. for focusing on spatial character structure information, and the UPT (Unary-Pairwise Transformer) algorithm proposed by Frederic Z Zhang et al. for unifying unary representation and pairwise representation. The two-stage method decouples the two tasks of person detection and relationship classification, allowing each stage to focus on its own task and produce good results in each stage.

[0005] However, both the one-stage and two-stage methods require a large amount of balanced training data, and the training process often consumes a lot of computing resources.

[0006] Vision-language pre-training is a very basic work in the field of artificial intelligence. In recent years, thanks to the introduction of the attention mechanism, this field has made rapid progress. Among them, CLIP (Contrastive Language-Image Pre-Training) is a neural network model trained on a large number of <image, text> pairs. It has strong zero-shot learning capabilities in tasks such as image classification and has been proven to be effective in many downstream tasks. For example, the Clip-adapter algorithm proposed by Peng Gao et al. only requires adapters to be added to the last few layers of the model, and the Adaptformer algorithm proposed by Shoufa Chen et al. inserts learnable sub-networks into each module of the model. However, these methods do not allow the model to pay attention to the spatial relationship between pairs, so they are not suitable for the task of human interaction detection. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention proposes a highly efficient and adaptive person interaction detection method and system based on concept-guided memory.

[0008] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0009] An efficient and adaptive image-based person interaction detection method based on concept-guided memory is used to detect all people and objects in an image and predict the interaction relationship between them. It includes the following two operating modes:

[0010] (1) Training-free mode: Using pre-trained object detectors and multimodal models, we build a multi-branch concept-guided person interaction memory module to simultaneously store domain-specific visual knowledge and domain-agnostic semantic knowledge guided by person interaction concepts. This allows the model to complete the person interaction detection task with only a small number of samples and achieve competitive performance.

[0011] (2) Fine-tuning mode: A lightweight adapter based on the attention mechanism is used to incorporate instance-level prior knowledge into the visual encoder in the model. At the same time, the knowledge stored in the person interaction memory module is updated to obtain more spatially sensitive pairwise relationship features, making the prediction of person interaction relationships more accurate.

[0012] Furthermore, step (1) uses a pre-trained object detector DETR to detect all people and objects in the image, and uses a multimodal pre-training model CLIP (Contrastive Language-Image Pretraining) to extract features for the detected target areas and language descriptions of different character interaction categories.

[0013] Furthermore, the domain-specific visual knowledge of step (1) requires features of three target regions, namely, the region of a person, the region of an object, and the union region consisting of the union of the above two regions.

[0014] Furthermore, the character interaction memory module in step (1) contains three branches, namely, the instance-centric branch, the interaction-aware branch, and the semantic branch. The instance-centric branch stores a fine-grained feature set centered on people and objects, i.e., the features of the human region and the object region; the interaction-aware branch stores features containing background information of the character interaction scene, i.e., the features of the human and object union region. The features stored in these two branches constitute the specific domain visual knowledge. In addition, the semantic branch stores various character interaction semantic feature sets with generalization ability, i.e., the universal domain semantic knowledge.

[0015] Furthermore, the instance-centric branch and interaction-aware branch of step (1) are represented by a dictionary-like <key, value> (<key,value> ) structure, with the corresponding extracted features as keys and the corresponding labels as values; the semantic branch is a classifier initialized with the semantic encoding of the interactions between the characters as weights.

[0016] Furthermore, step (2) is based on the initialization of the concept-guided character interaction memory module constructed in step (1), so that the keys of the instance center branch and the interaction awareness branch, as well as the weights of the semantic branch classifier become three learnable parameters, while keeping the values ​​of the instance center branch and the interaction awareness branch fixed, so that the model can select the memory content to be emphasized or suppressed, that is, the three learnable parameters.

[0017] Furthermore, step (2) introduces a lightweight prior knowledge injection adapter (Instance-awareAdapter) to integrate instance-level prior knowledge into the early spatial feature map of the visual encoder, so that the model can pay more attention to the spatial position characteristics of people and objects and better serve the pairwise relationship prediction task.

[0018] Furthermore, the instance-level prior knowledge of step (2) includes: semantic features, which are the semantic encoding of the instance, providing language prior knowledge to the visual encoder, which can prompt it what types of interactions the current object may contain; spatial position features, which provide geometric spatial position information of the instance in the image, allowing the visual encoder to distinguish different instance types in a fine-grained manner; and confidence scores, which reflect the uncertainty of the current instance judgment.

[0019] Furthermore, in order to make the model lightweight, step (2) first maps the dimensions and feature representations of the prior knowledge into the same shared low-dimensional space through low-dimensional mapping, and then integrates the prior knowledge into the visual encoder through the attention mechanism, thereby obtaining better visual encoding features of the character.

[0020] Furthermore, step (2) compares the cosine similarity of the key of the character's visual encoding features with the instance center branch and the interactive consciousness branch in the character interaction memory module, and uses them as weights to weight the values ​​of each branch to obtain the prediction results of the above two branches. At the same time, the character's visual encoding features are passed through the classifier of the semantic branch to obtain the prediction result of the branch. Finally, the prediction results of different branches are fused in a weighted average manner to obtain the final interaction type prediction result. The model uses focal loss to alleviate the imbalance problem between positive and negative samples.

[0021] An efficient and adaptive picture-person interaction detection system based on concept-guided memory using the above method comprises:

[0022] The training-free module uses pre-trained object detectors and multimodal models to build a multi-branch, concept-guided person interaction memory module to store both domain-specific visual knowledge and general domain semantic knowledge, allowing the model to complete task interaction detection tasks with only a small number of samples;

[0023] The fine-tuning module uses a lightweight adapter based on the attention mechanism to incorporate instance-level prior knowledge into the visual encoder in the model, and at the same time updates the knowledge stored in the character interaction memory module to obtain more spatially sensitive paired character features, thereby improving the accuracy of character interaction detection by learning only a small amount of parameters.

[0024] The beneficial effects of the present invention are:

[0025] On the one hand, the present invention can make full use of the existing pre-trained models, so that the model can simultaneously utilize the specific domain visual knowledge and the general domain semantic knowledge guided by the concept of human interaction, and complete the human interaction detection task without training and only using a small number of labeled samples; on the other hand, in order to efficiently adapt to the new domain, the present invention proposes to incorporate prior knowledge into the model, so that the model can be efficiently transferred to the new domain through lightweight training. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the efficient and adaptive image-person interaction detection method of the present invention.

[0027] Figure 2 It is a schematic diagram of the lightweight adapter structure of the present invention that can incorporate instance-level prior information. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] The present invention provides an efficient and adaptive method for detecting human interaction in pictures, the process of which is as follows: Figure 1 As shown, it specifically includes the following modes:

[0030] (1) Training-free mode

[0031] In the training-free mode, the model uses the pre-trained object detector DETR to detect all people and objects in the image, and uses the pre-trained CLIP to extract features for the target area and the language description of different types of human interactions. The area of ​​people and objects detected by the object detector can be denoted by b h ,b o , for each pair of people and objects, the union area can be recorded as b uThe features extracted by CLIP for the human region, object region, and human union region can be denoted as f h ,f o ,f u The present invention connects f h and f o to form object-centric fine-grained features f IC , directly put f u As a feature containing the background information of the scene where people interact IA .

[0032] For instance central branch <F IC ,L IC >, where F IC and L IC Respectively represent the key and value in the dictionary structure in this branch. The present invention extracts the instance-centric feature f from the training set images. IC F stored in this branch IC , convert the corresponding label into a multi-hot vector and store it in L in this branch IC .

[0033] For the interactive consciousness branch <F IA ,L IA >, similarly, F IA ,L IA Respectively represent the key and value of the dictionary structure in the branch. The present invention uses the feature f that is rich in interactive information and the spatial position relationship of the characters u Stored in F IA , convert the corresponding label into a multi-hot vector and store it in L IA .

[0034] For the semantic branch, the present invention aims to use language common sense to improve the performance of interaction category prediction. The classifier of this branch can be denoted as W T In order to construct W T First, we manually design a prompt sentence for each character interaction category, such as "a picture of a person holding an object". The prompt sentence will be sent to the CLIP language encoder to obtain its encoding vector in the visual-language common feature space. This vector can be used as the representative vector of this action category and stored in W as the classifier weight. T The semantic branch can be easily generalized to new categories of human interaction by utilizing the representation of the common visual-linguistic feature space and adding semantic encodings of new categories.

[0035] In the test phase, for each person pair in the test image, the features of each region are extracted to form f IC ,f IAEtc., and then use the following formula to obtain the interaction category prediction of the character pair:

[0036]

[0037]

[0038] where γ IC , γ IA , γ T are the weight coefficients of the three branches respectively, Predict the interaction category corresponding to the current person and object.

[0039] (2) Fine-tuning mode

[0040] In fine-tuning mode, the present invention integrates prior knowledge into the visual encoder through a lightweight adapter. The structure of the lightweight adapter is as follows Figure 2 shown.

[0041] Prior knowledge consists of three parts: semantic features, which are the semantic encoding of the instance, providing language prior knowledge to the visual encoder, which can prompt it what types of interactions the current object may contain; spatial position features, which provide geometric spatial position information of the instance in the image, allowing the visual encoder to distinguish different instance types in a fine-grained manner; and confidence scores, which reflect the uncertainty of the current instance judgment.

[0042] The instance-level adapter is a lightweight sub-module that can be inserted into each layer of the visual encoder to help the model learn to integrate prior knowledge from objects detected by the target detector, so that the visual encoder can better serve downstream tasks.

[0043] Further, use represents the i-th layer feature map of the visual encoder, where H′W′ represents the size of the feature map and d represents the spatial dimension of the feature map. In order to reduce the number of parameters and computational cost, a low-dimensional mapping matrix is ​​used and high-dimensional mapping matrix It is possible to compress the feature matrix into a low-dimensional space or map it back to a high-dimensional space, where d′<<d. Then, the multi-head cross attention mechanism (MHSA) is used to implement prior knowledge injection:

[0044] H t =MLP p (P t ) (3)

[0045] X′ i =MHSA((X i ·W down ),Ht ,H t )·W up (4)

[0046] Among them, X′ i represents prior knowledge, Represents N extracted from the image t Instance-level prior knowledge, p t = {b t ,c t ,e t}, given by the instance's bounding box coordinates b t , the confidence level c of the instance t and the semantic encoding of the instance t Composition. t Represents the prior knowledge through the fully connected network MLP p The hidden state after mapping is in the same feature space as the feature map of the visual encoder after the dimension reduction, where H t It is the Key and Value in the multi-head attention mechanism, and the feature map X with reduced dimensions i ·W down It is the Query in the multi-head attention mechanism.

[0047] like Figure 1 As shown in the figure, during the training process, the weights of the dark gray modules of the model are fixed, and the weights of the light gray modules will be updated. The model uses focal loss To maximize the prediction accuracy, update the weights:

[0048]

[0049]

[0050] in For a given input image, is the target detector, f is the visual encoder, ε is the visual feature encoded by the image, is the character interaction memory module, g is the character interaction category prediction function defined by formulas (1) and (2), is the true interaction label, θ is the parameter to be optimized, and θ * are the parameters finally optimized.

[0051] The following experimental results show that compared with existing methods, the task interaction detection method of the present invention can achieve higher accuracy with less training time.

[0052] This embodiment uses the HICO-DET dataset. The dataset consists of 47,776 images, of which the training set contains 38,118 images and the test set contains 9,658 images. Its 600 character interaction categories are composed of 117 action categories and 80 object categories. There are many types of interactions involved, and the same character pair may have multiple interaction relationships. Therefore, it is very challenging and helps to prove the effectiveness of this method.

[0053] Existing method 1: The method proposed by Bumsoo Kim et al. in the paper “Hotr: End-to-end human-object interaction detection with transformers.”

[0054] Existing method 2: The method proposed by Frederic Z Zhang et al. in the paper “Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer.”

[0055] Existing method 3: The method proposed by Yue Liao et al. in the paper “Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection”.

[0056] The present invention: the method of this embodiment.

[0057] The experiment uses the mAP indicator to evaluate the accuracy of human interaction detection. This indicator measures the positioning accuracy and relationship prediction accuracy of people and objects at the same time. A group of human interactions is considered to be successfully detected only when the positioning is accurate, the intersection of the detected bounding box and the true bounding box is greater than 0.5, and the interaction relationship is correctly predicted.

[0058] It can be seen from Table 1 that the present invention achieves better human interaction detection results while using the least amount of learnable parameters. Although some existing methods also utilize the capabilities of pre-trained models, they have not established an efficient human interaction detection system, and the model still needs to update a large number of learnable parameters.

[0059] Table 1. Comparative experimental results with existing methods

[0060]

[0061]

[0062] Based on the same inventive concept, another embodiment of the present invention provides an efficient image-person interaction detection system, which includes:

[0063] A training-free module is used to build a multi-branch concept-guided person interaction memory module using pre-trained object detectors and multimodal models. The person interaction memory module stores both domain-specific visual knowledge and general domain semantic knowledge, so that the person interaction detection task can be completed with only a small number of samples and competitive performance can be achieved;

[0064] The fine-tuning module is used to utilize a lightweight adapter based on the attention mechanism to incorporate instance-level prior knowledge into the visual encoder, and at the same time update the knowledge stored in the character interaction memory module to obtain more spatially sensitive pairwise relationship features, making the prediction of character interaction relationships more accurate.

[0065] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0066] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0067] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An efficient and adaptive method for detecting human interaction in pictures based on concept-guided memory, characterized in that: The following modes are included: Training-free mode: Using pre-trained object detectors and multimodal models, a multi-branch concept-guided person interaction memory module is constructed. The person interaction memory module stores both domain-specific visual knowledge and general domain semantic knowledge, so that the person interaction detection task can be completed with only a small number of samples and competitive performance can be achieved; Fine-tuning mode: A lightweight adapter based on the attention mechanism is used to incorporate instance-level prior knowledge into the visual encoder, while updating the knowledge stored in the character interaction memory module to obtain more spatially sensitive pairwise relationship features, making the prediction of character interaction relationships more accurate.

2. The method according to claim 1, characterized in that The training-free mode uses the pre-trained object detector DETR to detect all people and objects in the image, and uses the pre-trained CLIP to extract features for the language description of the target area and different types of human interaction; The human and object regions detected by the target detector are denoted as b h ,b o , for each pair of people and objects, the union area is denoted as b u ; The features extracted by CLIP for the human region, object region, and human union region are denoted as f h ,f o ,f u , link f h and f o Composed of object-centric fine-grained features f IC , directly put f u As the feature f containing the scene information of human interaction IA .

3. The method according to claim 1, characterized in that The character interaction memory module includes three branches, namely, an instance center branch, an interaction awareness branch and a semantic branch; the instance center branch stores a fine-grained feature set centered on people and objects, namely, features of the human area and the object area; the interaction awareness branch stores features containing background information of the character interaction scene, namely, features of the union area of ​​people and objects; the features stored in the instance center branch and the interaction awareness branch constitute specific domain visual knowledge; the semantic branch stores various character interaction semantic feature sets with generalization ability, namely, universal domain semantic knowledge.

4. The method according to claim 3, characterized in that For instance central branch <F IC ,L IC >, where F IC and L IC Represent the keys and values ​​in the dictionary structure in this branch, respectively, and the instance-centric feature f IC The F stored in this branch IC , the corresponding labels are converted into multi-hot vectors and stored in L in this branch IC ; For the interactive consciousness branch <F IA ,L IA >, features rich in interactive information and spatial position relationships between characters f u Stored in F IA , the corresponding labels are stored in L IA ; For the semantic branch, we manually design prompt sentences for each character interaction category, encode them using CLIP, and store them in the classifier W as classifier weights. T .

5. The method according to claim 3, characterized in that The fine-tuning mode compares the cosine similarity of the character's visual encoding features with the keys of the instance center branch and the interactive awareness branch in the character interaction memory module, and uses them as weights to weight the values ​​of each branch to obtain the prediction results of the two branches. At the same time, the character's visual encoding features are passed through the classifier of the semantic branch to obtain the prediction result of the branch. Finally, the prediction results of different branches are fused in a weighted average manner to obtain the final interaction type prediction result.

6. The method according to claim 1, characterized in that The instance-level prior knowledge includes: semantic features, which are semantic encodings of instances, providing language prior knowledge for the visual encoder, suggesting which types of interactions the current object may contain; spatial position features, which provide geometric spatial position information of instances in the image, allowing the visual encoder to distinguish different instance types in a fine-grained manner; and a confidence score, which reflects the uncertainty of the current instance judgment.

7. The method according to claim 1, characterized in that The lightweight adapter is inserted into each layer of the visual encoder to help learn and integrate prior knowledge from objects detected by the target detector; Represents the i-th layer feature map of the visual encoder, where H′W′ represents the size of the feature map, d represents the spatial dimension of the feature map, and a low-dimensional mapping matrix is ​​used. and high-dimensional mapping matrix Compress the feature matrix into a low-dimensional space or map it back to a high-dimensional space, where d′<<d; then use the multi-head attention mechanism MHSA to implement prior knowledge injection: H t =MLP p (P t ) X′ i =MHSA((X i ·W down ),H t ,H t )·W up Among them, X′ i represents prior knowledge, Represents N extracted from the image t Instance-level prior knowledge, p t = {b t ,c t ,e t }By the bounding box coordinates of the instance b t , the confidence level c of the instance t and the semantic encoding of the instance t Composition, H t Representing prior knowledge through a fully connected network MLP p Hidden state after mapping.

8. The method according to claim 7, characterized in that Through focal loss To maximize the prediction accuracy and update the weights: in For a given input image, is the target detector, f is the visual encoder, and ε is the visual feature encoded by the image is the character interaction memory module, g is the character interaction category prediction function, is the true interaction label, θ is the parameter to be optimized, and θ * are the parameters finally optimized.

9. A weakly supervised video temporal positioning system using the method of any one of claims 1 to 8, characterized in that: include: A training-free module is used to build a multi-branch concept-guided person interaction memory module using pre-trained object detectors and multimodal models. The person interaction memory module stores both domain-specific visual knowledge and general domain semantic knowledge, so that the person interaction detection task can be completed with only a small number of samples and competitive performance can be achieved; The fine-tuning module is used to utilize a lightweight adapter based on the attention mechanism to incorporate instance-level prior knowledge into the visual encoder, and at the same time update the knowledge stored in the character interaction memory module to obtain more spatially sensitive pairwise relationship features, making the prediction of character interaction relationships more accurate.

10. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 8.

Images