Method and apparatus for human-object interaction detection based on affordance-scene knowledge cues

Abstract

This invention discloses a method and apparatus for detecting human-object interactions based on availability-scene knowledge prompts, which can improve the recognition ability in real-world scenarios and predict more accurate HOI detection results. The method includes: (1) constructing an availability-scene knowledge graph (ASKG); (2) building a human-object interaction detection model: in the training phase, the input is an RGB image. In the first phase, people and objects are detected, scene categories are identified, and original features are encoded; in the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance-aware features, the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene-aware features, and finally the results are output through the HOI recognition module; (3) performing human-object interaction detection on the image.

Description

Technical Field

[0001] This invention relates to the technical fields of computer vision and human-object interaction detection, and in particular to a human-object interaction detection method based on availability-scene knowledge prompts, and a human-object interaction detection device based on availability-scene knowledge prompts. Background Technology

[0002] Human-object interaction (HOI) detection requires locating and identifying interactions between pairs of people and objects in an image. HOI is typically defined as...<human,action,object> Triples, where human and object representations are described using labeled bounding boxes, and human actions are defined as interactions with objects. HOI detection addresses the human-centered object interaction problem and is helpful for many downstream tasks such as human activity understanding, human-computer interaction, and visual knowledge question answering.

[0003] Thanks to the powerful feature representation capabilities of deep CNNs, HOI detection methods in recent years have been designed based on deep learning. These methods are generally divided into two types: two-stage and one-stage. In two-stage methods, the first stage involves using a pre-trained object detector to locate people and objects in an image, while the second stage focuses on predicting the interactions between each given person-object pair. Compared to two-stage methods, one-stage methods typically employ a robust representation learning model that can perform person and object detection and interaction prediction in parallel. Recently, Transformer models have demonstrated superior capabilities in many vision tasks, including HOI detection. HOI-TR was the first to attempt to replace carefully designed components with a simple Transformer encoder-decoder architecture, which serves as the backbone for feature extraction and the detector for HOI prediction. Since then, many Transformer-based methods have emerged and achieved excellent performance.

[0004] While these deep HOI models have achieved satisfactory results, HOI detection methods, regardless of their architecture, still need to address the problem of the natural long-tail distribution of HOI classes. Due to the complexity of scenes and the diversity of HOI categories, it is difficult to detect multiple interactions within a scene. Furthermore, there is the zero-shot HOI detection problem, where classes appearing in the test set do not appear in the training set, making HOI detection a significant challenge. Many efforts have attempted to address the long-tail distribution and zero-shot problem by integrating external knowledge and transfer learning. However, most of these methods only utilize a limited label space to extract knowledge from labels, lacking sufficient external knowledge. In this situation, if visual information is insufficient and there are not enough external knowledge cues to guide the HOI detection model, it is difficult to improve the performance of HOI recognition. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, the technical problem to be solved by the present invention is to provide a human-object interaction detection method based on availability-scene knowledge prompts, which can improve the recognition ability in real-world scenarios and predict more accurate HOI detection results.

[0006] The technical solution of this invention is: a human-object interaction detection method based on availability-scene knowledge cues, comprising the following steps:

[0007] (1) Constructing an Availability-Scene Knowledge Graph (ASKG): First, extract the object corresponding to each image and the related actions from the dataset annotations. The images and their annotations are from the Human-Object Interaction Detection (HOI) datasets HICO-DET and V-COCO. The annotations are composed of triples.<human,action,object> The system is composed of three sets of triplets: a scene classification network is used to identify scene categories and attributes for each image, and a contextual knowledge graph is constructed.<scene,has,object> ,<object,has,affordance> and<scene,has,property> In this way, different objects, scenes and the relationships between them are linked; the final ASKG is obtained by filtering out identical triples. The visibility-scene knowledge graph retrieves images by object and scene, generates image-related subgraphs, and extracts specific knowledge.

[0008] (2) Building a human-object interaction detection model: In the training phase, the input is an RGB image. The first phase detects people and objects, identifies scene categories and encodes original features. In the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance perception features, the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene perception features, and finally the HOI recognition module outputs the results.

[0009] (3) Perform human-object interaction detection on the image.

[0010] This invention first constructs a large-scale Availability Scene Knowledge Graph (ASKG). Based on the domain of image information, the knowledge in the ASKG is divided into two categories: knowledge related to the availability of object instances and knowledge related to the scene. Subsequently, a prior knowledge embedding module based on the ASKG extracts specific availability and scene knowledge from the input image. Since this knowledge corresponds to images of different granularities, this invention proposes instance-domain adaptive fusion and scene-domain adaptive fusion, allowing visual features to fully absorb knowledge cues. The encoded features and knowledge embeddings from these two different domains are ultimately used for human-object interaction detection in the input image. Therefore, it can improve recognition capabilities in real-world scenes and predict more accurate HOI detection results.

[0011] A human-object interaction detection device based on availability-scene knowledge cues is also provided, the device comprising:

[0012] The building module, configured to construct the Availability-Scene Knowledge Graph (ASKG), first extracts the object corresponding to each image and its associated action from the dataset annotations. The images and their annotations are from the Human-Object Interaction Detection (HOI) datasets HICO-DET and V-COCO. The annotations are composed of triples.<human,action,object> The system is composed of three sets of triplets: a scene classification network is used to identify scene categories and attributes for each image, and a contextual knowledge graph is constructed.<scene,has,object> ,<object,has,affordance> and

[0013] <scene,has,property> In this way, different objects, scenes and the relationships between them are linked; the final ASKG is obtained by filtering out identical triples. The visibility-scene knowledge graph retrieves images by object and scene, generates image-related subgraphs, and extracts specific knowledge.

[0014] The module is configured to build a human-object interaction detection model: During the training phase, the input is an RGB image. The first phase detects people and objects, identifies scene categories, and encodes original features. In the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance-aware features, and the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene-aware features. Finally, the HOI recognition module outputs the results.

[0015] The detection module is configured to perform human-object interaction detection in images. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the construction process of the Availability-Scenario Knowledge Graph (ASKG) according to the present invention is shown.

[0017] Figure 2 The ASK-HOI framework diagram is shown.

[0018] Figure 3 The visual text multimodal attention module is shown.

[0019] Figure 4 Examples of visualizations of detection results on the HICO-DET and V-COCO datasets are shown. Detailed Implementation

[0020] To address the aforementioned issues, it is proposed to leverage richer textual knowledge as cues to improve HOI detection performance. HOI detection relies on a deep understanding of real-world human scenes, which are complex and diverse yet contain rich and valuable information. Current research primarily focuses on utilizing HOI tags or image titles to enhance HOI detection performance, but it hasn't fully utilized the information within the images. This paper further refines image information into two fields: the object instance domain and the scene domain. The former provides local object-related information, while the latter provides global scene context information. Based on these two granularities of image information, corresponding textual knowledge for the object instance domain and scene domain is proposed. It is suggested to use scene categories and their attributes in the image as scene-related knowledge cues. Furthermore, human-object pair interactions are closely related to object affordance (i.e., the interactive capabilities of objects), reflecting the "actions" in the HOI triples. Therefore, introducing object affordance knowledge can provide more direct cues for HOI detection. In recent years, text-image contrastive learning has made significant progress in various visual tasks. The goal of image-text contrastive learning is to align their features and obtain supervised signals from the text. Inspired by this, it is proposed to embed this textual knowledge into the HOI detector. This allows the HOI model to learn better discriminative features, thereby improving detection accuracy.

[0021] Based on this, a large-scale Availability-Scene Knowledge Graph (ASKG) was first constructed. This graph consists of three sets of triples, namely...<scene,has,object> ,<object,has,affordance> and<scene,has,property> The first set of triples represents all objects that may exist in a specific scene. The second set of triples represents the availability of objects that can be realized in that scene. The last set of triples represents knowledge of the attributes possessed by the scene. ASKG contains rich semantic information; the first and third sets of triples correspond to scene-domain related knowledge, while the second set of triples corresponds to object instance-domain related functional knowledge. Scene knowledge can provide global contextual information for visual features, while object availability knowledge can serve as direct cues for action prediction in HOIs. Especially for the long-tailed distribution and zero-shot problem of HOIs, ASKG can provide additional learnable semantic features for HOI detection models.

[0022] Based on ASKG, a novel two-stage HOI detection model, ASK-HOI, is proposed. The first stage of ASK-HOI utilizes three pre-trained models to detect people and objects, classify the scene, and extract the original features of the input image. The second stage of ASK-HOI consists of four modules that utilize the results of the first stage for HOI detection. First, there is a Prior Knowledge Embedding (PKEM) module, which extracts scene- and object-specific subgraphs from the input image using ASKG. The relevant knowledge in these subgraphs is mapped to a semantic embedding space through a pre-trained image-text encoder. Since this knowledge corresponds to different granularities of domains in the image, a perceptual domain-adaptive text-visual matching strategy is proposed to fuse knowledge and visual features of different granularities. Specifically, an Instance Domain Adaptive Fusion Module (IFAFM) and a Scene Domain Adaptive Fusion Module (SFAFM) are proposed to encode high-quality visual features at different levels. IFAFM fuses knowledge embedding with human, object, and human-object spatial features to encode instance domain perceptual features. Here, only the visibility knowledge of objects in the scene is extracted, excluding scene attributes. Since IFAFM only fuses local instance features of people and objects, other scene knowledge may become noise. In contrast, SFAFM encodes visual features from the global scene domain. Therefore, it embeds all scene knowledge and uses image-text dependencies to refine visual features. To achieve full cross-model fusion of textual knowledge and visual features, a general visual-text multi-model attention plugin (VT-Attention) is proposed, which can be used for fusion of arbitrary visual and textual features in IFAFM and SFAFM. Finally, a fourth module, the HOI recognition module (HOIRM), is proposed. Instance-aware features are input from IFAFM, scene-aware features from SFAFM, and knowledge hints from PKEM are embedded into HOIRM. It utilizes multi-head self-attention and cross-attention to fully fuse visual and semantic features, thereby predicting high-performance HOI detection results.

[0023] like Figure 1 As shown, this human-object interaction detection method based on availability-scene knowledge cues includes the following steps:

[0024] (1) Constructing an Availability-Scene Knowledge Graph (ASKG): First, extract the object corresponding to each image and the related actions from the dataset annotations. The images and their annotations are from the Human-Object Interaction Detection (HOI) datasets HICO-DET and V-COCO. The annotations are composed of triples. <human,

[0025] The system consists of `action, object` elements; simultaneously, for each image, a scene classification network is used to identify the scene category and scene attributes; a contextual knowledge graph is constructed, consisting of three sets of triples, namely...<scene,has,object> ,<object,has,affordance> and <scene,

[0026] The form of `has, property>` is used to connect different objects, scenes, and the relationships between them; the final ASKG is obtained by filtering out identical triples. The visibility-scene knowledge graph retrieves images by object and scene, generates image-related subgraphs, and extracts specific knowledge.

[0027] (2) Building a human-object interaction detection model: In the training phase, the input is an RGB image. The first phase detects people and objects, identifies scene categories and encodes original features. In the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance perception features, the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene perception features, and finally the HOI recognition module outputs the results.

[0028] (3) Perform human-object interaction detection on the image.

[0029] This invention first constructs a large-scale Availability Scene Knowledge Graph (ASKG). Based on the domain of image information, the knowledge in the ASKG is divided into two categories: knowledge related to the availability of object instances and knowledge related to the scene. Subsequently, a prior knowledge embedding module based on the ASKG extracts specific availability and scene knowledge from the input image. Since this knowledge corresponds to images of different granularities, this invention proposes instance-domain adaptive fusion and scene-domain adaptive fusion, allowing visual features to fully absorb knowledge cues. The encoded features and knowledge embeddings from these two different domains are ultimately used for human-object interaction detection in the input image. Therefore, it can improve recognition capabilities in real-world scenes and predict more accurate HOI detection results.

[0030] Preferably, in step (1), during the construction of triples...<object,has,affordance> At that time, extract some availability of the object.

[0031] Preferably, in step (1), the availability-scene knowledge graph includes: 365 scene categories, 102 scene attributes, 80 object categories, 117 function categories, and a total of 17,806 knowledge entries.

[0032] Preferably, step (2) includes the following sub-steps during the training phase:

[0033] (2.1) Encode the knowledge embedding of object availability and image scene. Based on the scene category in the classification, filter out the subgraph of all objects, object availability and scene attributes contained in the scene from the ASKG. Filter the objects, compare the detected target category with the scene subgraph, and remove the knowledge that does not contain the target to obtain the knowledge specific to the input image. This knowledge includes: scene category, scene attribute, object category and object availability.

[0034] (2.2) Extract the regions of people and objects, as well as the features within their joint spatial regions, and align them using convolutional layers respectively; then, perform instance features on people. Instance characteristics of objects Human-object joint regional characteristics Summation yields the fused instance domain features.

[0035]

[0036] (2.3) Embed the scene knowledge and object availability knowledge output by PKEM into With visual features Fusion yields visual availability-scene knowledge features. Simultaneously embed the coded title generated by PKEM By introducing and fusing subtitle embedding with visual features, visual subtitle knowledge features are calculated. Visual availability-scene features and visual caption knowledge characteristics

[0037] Channel concatenation is performed, and then a 1×1 convolutional layer is used to reduce the number of channels to obtain the final scene-aware features.

[0038] (2.4) Instance-aware features Availability-Scene Knowledge Embedded Features Scene-aware features As input for human-object interaction detection, the final HOI triple is output as the action category.

[0039] Preferably, in step (2.1), the availability knowledge corresponds to the instance domain information of the image, the availability-scene knowledge corresponds to the global scene domain information of the image, the image caption is used as the global knowledge of the input image, and the HOI triplet label and object label are converted into text descriptions as image titles;

[0040] The extracted knowledge cues are projected into the feature space using the contrastive language image pre-trained model CLIP, and then the text encoder f of CLIP is used. T(·) Encode knowledge availability-scenario knowledge using a text encoder f. T (·) First, the text sequence containing n words is converted into word embeddings. Then, the embeddings of each word are summed to obtain the final availability-context knowledge embedding features.

[0041]

[0042] According to formula (1), the availability knowledge is converted into semantic embeddings respectively. Transferring image captioning knowledge to semantic embedding

[0043] Preferably, in step (2.2), text embedding and fused instance domain visual features Input, calculate the final instance domain-aware features according to formulas (3)-(6)

[0044]

[0045]

[0046]

[0047]

[0048] in,

[0049] These are learnable weights. Norm represents L2 normalization, and SA and CA represent self-attention and cross-attention, respectively. They follow a unified formula, namely...

[0050]

[0051] Preferably, in step (2.4), by calculation and The dot product between them is then used with the softmax function to... and The input is fed into the cross-attention block to estimate the normalized scalar cross-attention matrix, and the attention matrix and... Calculate the new feature representation The calculation method is as follows:

[0052]

[0053] Among them, W q W k and W vAll are learnable linear transformation weights;

[0054] The results are input into a feedforward network FFN, which consists of two fully connected layers with a nonlinear mapping GELU. After Nl iterations, the fully fused features F vt The input is fed into an MLP with two fully connected layers, which outputs the final predicted HOI action prediction score.

[0055] Preferably, step (3) includes the following sub-steps:

[0056] (3.1) The input is an RGB image;

[0057] (3.2) The RGB image detects the instance features of people, objects, and the joint region of people and objects through the object detection model, obtains the scene category of the image through the scene classification network, and obtains the original visual features of the image through the backbone network;

[0058] (3.3) Extract image-specific object availability knowledge, scene knowledge, and image caption knowledge, and embed them into the feature space;

[0059] (3.4) The instance features are fused and then fused with the availability knowledge embedding through VT-Attention to output instance-aware features;

[0060] (3.5) The original visual features of the image are fused with the availability-scene knowledge embedding and the caption knowledge embedding respectively, and their channels are concatenated. After convolution to reduce the number of channels, the scene-aware features are output.

[0061] (3.6) Embed instance-aware features, scene-aware features, and availability-scene knowledge into the input to predict the action category, thereby obtaining the HOI triplet, which is then combined with the detection boxes of people and objects obtained by the object detection model as the final HOI detection result.

[0062] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. When executed, the program includes the steps of the methods of the above embodiments. The storage medium can be ROM / RAM, magnetic disk, optical disk, memory card, etc. Therefore, corresponding to the method of the present invention, the present invention also includes a human-object interaction detection device based on availability-scene knowledge prompts. This device is typically represented in the form of functional modules corresponding to the steps of the method. The device includes:

[0063] The building module, configured to construct the Availability-Scene Knowledge Graph (ASKG), first extracts the object corresponding to each image and its associated action from the dataset annotations. The images and their annotations are from the Human-Object Interaction Detection (HOI) datasets HICO-DET and V-COCO. The annotations are composed of triples.<human,action,object> The system is composed of three sets of triplets: a scene classification network is used to identify scene categories and attributes for each image, and a contextual knowledge graph is constructed.<scene,has,object> ,<object,has,affordance> and

[0064] <scene,has,property> In this way, different objects, scenes and the relationships between them are linked; the final ASKG is obtained by filtering out identical triples. The visibility-scene knowledge graph retrieves images by object and scene, generates image-related subgraphs, and extracts specific knowledge.

[0065] The module is configured to build a human-object interaction detection model: During the training phase, the input is an RGB image. The first phase detects people and objects, identifies scene categories, and encodes original features. In the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance-aware features, and the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene-aware features. Finally, the HOI recognition module outputs the results.

[0066] The detection module is configured to perform human-object interaction detection in images.

[0067] The beneficial technical effects of the present invention are demonstrated through the following verification.

[0068] This invention has conducted comparative experiments with the latest HOI detection methods and achieved the best results. In addition, ablation experiments were conducted on the proposed prior knowledge embedding module, instance domain adaptive fusion module and scene domain adaptive fusion module.

[0069] 1. Experimental data: This invention was tested on the widely used HOI detection datasets HICO-DET and V-COCO.

[0070] 2. Measurement metrics: The mean accuracy (mAP) of the overall category is used for evaluation.

[0071] 3. Experimental Setup: The Adam optimizer was selected to optimize the objective function. In all experiments, the learning rate was set to 0.0001, the batch size to 32, and a total of 100 epochs were trained. A domain adaptive encoder network was added for joint training after 10 epochs. The entire model was built using Python 3.6 and the PyTorch 1.7.1 deep learning framework, and trained using an NVIDIA RTX 3090 24G GPU.

[0072] 4. Comparison with State-of-the-art Methods: Table 1 shows the quantitative comparison results of the proposed method with selected state-of-the-art methods on the HICO-DET and V-COCO datasets. Compared to single-stage methods that utilize only visual features (such as MUREN, QPIC, and HOTR), this method has a significant advantage. Currently, the best-performing text-based single-stage methods, HOICLIP and GEN-VLKT, both utilize CLIP-encoded textual knowledge to improve HOI recognition accuracy. Using the same text embedding, this method outperforms these methods by 1.52 mAP and 2.46 mAP, respectively, in the complete HOI category. For two-stage methods, ViPLO currently performs best, also utilizing textual information as prior knowledge. In comparison, for the complete HOI category of the HICO-DET dataset, this method outperforms ViPLO by 1.27 mAP, and for scenario 1 of the V-COCO dataset, it outperforms by 8.0 mAP. The advantages of this method are more pronounced than other two-stage methods that do not utilize textual knowledge. Notably, the improvement is more significant in rare categories. The reason is that, in addition to using image caption text as used in most methods, specific scene and object availability knowledge is added to the images. This knowledge effectively supplements the information the model needs for rare categories, and can better handle the long-tail distribution problem of HOI categories. For the V-COCO dataset, the results for scenarios 1 and 2 are reported in Table 1, respectively. This method achieves the best mAP metric, which is 68.9 and 72.1, respectively.

[0073] Table 1 shows a quantitative comparison with state-of-the-art methods on the HICO-DET and V-COCO datasets (sorted in ascending order of mAP; bold text indicates best performance, underline indicates second-best).

[0074]

[0075]

[0076] 5. Visualization of experimental results: In Figure 4The document visualizes some of the detection results of this invention on the HICO-DET and V-COCO datasets. If a single entity in an image contains multiple interactions, they are separated by commas ",". If multiple entities have different interactions, they are separated by semicolons ";". If they have the same interaction, parentheses "()" are used to indicate multiple interactions and their scores. The results show that this invention achieves satisfactory HOI detection results in various indoor and outdoor scenes. The proposed ASKG can provide the HOI detector with more knowledge about the image scene and specific objects, enabling the model to more accurately identify human-object interactions. Especially for many-to-many HOI detection, it can be seen that this invention achieves optimal HOI detection results.

[0077] 6. Ablation Experiments: To verify the impact of the main modules (PKEM, IFAFM, SFAFM) in this invention, extensive ablation studies were conducted on the HICO-DET and V-COCO datasets. Specific results are shown in Tables 2 and 3, with bold text indicating optimal performance and underlined text indicating suboptimal performance.

[0078] Table 2 shows the ablation experiments of individual modules in the HICO-DET and V-COCO datasets.

[0079]

[0080] Table 3 shows ablation experiments with different knowledge embeddings in a single module of the HICO-DET and V-COCO datasets.

[0081]

[0082] First, PKEM was removed, meaning the complete model lacks object availability, scene, and image captioning knowledge. Table 2 shows that the accuracy of the model without PKEM is significantly lower compared to the complete model. Furthermore, the differences between availability knowledge (AK), scene knowledge (SK), and image captioning knowledge (ICK) were compared. As shown in Table 3, adding all knowledge provides a greater gain than adding only availability, scene, or captioning knowledge. Each part of the knowledge contributes rich information, helping the model encode distinctive features. The improvement resulting from the combination of these complementary knowledge pieces is even more significant.

[0083] Secondly, removing IFAFM resulted in a slight decrease in accuracy, as shown in Table 2, which validates the effectiveness of object instance domain information. Furthermore, the impact of knowledge guidance was compared in this module. The experimental results are shown in Table 3; adding knowledge significantly improved accuracy compared to not adding knowledge. This indicates that knowledge prompts have a positive impact on instance feature modeling. Additionally, the impact of different knowledge inputs on IFAFM was compared. Since IFAFM extracts specific instance field features of people and objects after removing background information, adding object availability knowledge (AK), i.e., object availability and object category, improved performance by 1.11 mAP. However, adding the entire availability-scene knowledge (ASK), which includes scene knowledge, to IFAFM resulted in a 0.50 mAP decrease in accuracy compared to AK. This indicates that global scene domain knowledge introduces noise into instance domain features. Therefore, using only availability knowledge is optimal in this module.

[0084] Finally, SFAFM was removed, and knowledge embedding was performed using only instance and object features combined with knowledge embedding. As shown in Table 2, the performance decreased by 1.44 mAP compared to the full model. This indicates that the global visual features of the image are not negligible. This module primarily discusses the impact of textual knowledge on model accuracy. As shown in Table 3, adding knowledge improved accuracy compared to using only the original image features. Furthermore, the input image captions were removed, as shown at the bottom of Table 3. In this module, removing image caption knowledge resulted in a 0.35 mAP decrease in accuracy compared to the full model, indicating that image captions also provide valuable prior information to the model. Additionally, the gain from caption knowledge was less than that from availability-scene knowledge. This further demonstrates that the constructed availability-scene knowledge plays a crucial role in improving the model's discriminative ability.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A human-object interaction detection method based on availability-scene knowledge prompts, characterized in that: The method includes the following steps: (1) Constructing the Availability-Scene Knowledge Graph (ASKG): First, extract the object corresponding to each image and the related actions from the dataset annotations. The images and their annotations are from the Human-Object Interaction Detection (HOI) datasets HICO-DET and V-COCO. The annotations are composed of triples.<human,action,object> The system is composed of three sets of triplets: a scene classification network is used to identify scene categories and attributes for each image, and a contextual knowledge graph is constructed.<scene,has,object> ,<object,has,affordance> and<scene,has,property> In this form, different objects, scenes, and the relationships between them are linked; the final ASKG is obtained by filtering out identical triples. The visibility-scene knowledge graph retrieves images by object and scene, generates image-related subgraphs, and extracts specific knowledge. (2) Building a human-object interaction detection model: In the training phase, the input is an RGB image. In the first phase, people and objects are detected, scene categories are identified, and original features are encoded. In the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance perception features, the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene perception features, and finally the results are output through the HOI recognition module. (3) Perform human-object interaction detection on the image; Step (2) includes the following sub-steps during the training phase: (2.1) Encode the knowledge embedding of object availability and image scene. Based on the scene category in the classification, filter out the subgraph of all objects, object availability and scene attributes contained in the scene from the ASKG. Filter the objects, compare the detected target category with the scene subgraph, and remove the knowledge that does not contain the target to obtain the knowledge specific to the input image. This knowledge includes: scene category, scene attribute, object category and object availability. (2.2) Extract the regions of people and objects, as well as the features of their joint spatial regions, and align them using convolutional layers respectively; then perform instance features on people. Instance characteristics of objects Human-object joint regional characteristics Summation yields the fused instance domain features. , （2）； (2.3) Embed the scene knowledge and object availability knowledge output by PKEM into With visual features Fusion yields visual availability-scene knowledge features. Simultaneously embed the coded title generated by PKEM. By introducing and fusing subtitle embedding with visual features, visual subtitle knowledge features are calculated. Visual availability-scene features and visual caption knowledge characteristics Channel concatenation is performed, and then a 1×1 convolutional layer is used to reduce the number of channels to obtain the final scene-aware features. ; (2.4) Instance-aware features Availability-Scene Knowledge Embedded Features Scene-aware features As input for human-object interaction detection, the final HOI triple is output as the action category.

2. The human-object interaction detection method based on availability-scene knowledge prompts according to claim 1, characterized in that: In step (1), when constructing the triple <object, has, affordance>, some affordances of the object are extracted.

3. The human-object interaction detection method based on availability-scene knowledge prompts according to claim 2, characterized in that: In step (1), the availability-scenario knowledge graph includes: 365 scenario categories, 102 scenario attributes, 80 object categories, 117 function categories, and a total of 17,806 knowledge entries.

4. The human-object interaction detection method based on availability-scene knowledge prompts according to claim 3, characterized in that: In step (2.1), the availability knowledge corresponds to the instance domain information of the image, and the availability-scene knowledge corresponds to the global scene domain information of the image. The image caption is used as the global knowledge of the input image, and the HOI triplet label and object label are converted into text descriptions as image titles. The extracted knowledge cues are projected into the feature space using the contrastive language image pre-trained model CLIP, and then the text encoder of CLIP is used. Encoding knowledge availability—context knowledge—using a text encoder. First, include n The text sequence of words is converted into word embeddings, and then the embeddings of each word are summed to obtain the final availability-context knowledge embedding features. , （1） According to formula (1), the availability knowledge is converted into semantic embeddings respectively. Transferring image captioning knowledge to semantic embedding .

5. The human-object interaction detection method based on availability-scene knowledge prompts according to claim 4, characterized in that: In step (2.2), text embedding and fused instance domain visual features Input, calculate the final instance domain-aware features according to formulas (3)-(6) , （3） （4） （5） （6） in, , These are learnable weights. Norm Indicates L2 normalization, SA and CA These represent self-attention and cross-attention, respectively, and they follow a unified formula, namely... .

6. The human-object interaction detection method based on availability-scene knowledge prompts according to claim 5, characterized in that: In step (2.4), by calculation and The dot product between them, then use softmax The function will , and The input is fed into the cross-attention block to estimate the normalized scalar cross-attention matrix, and the attention matrix and... Calculate the new feature representation The calculation method is as follows: （7） in, , and All are learnable linear transformation weights; The results are input into a feedforward network FFN, which consists of two fully connected layers with a nonlinear mapping GELU. After the next iteration, the fully fused features The input is fed into an MLP with two fully connected layers, which outputs the final predicted HOI action prediction score.

7. The human-object interaction detection method based on availability-scene knowledge prompts according to claim 6, characterized in that: Step (3) includes the following sub-steps: (3.1) The input is an RGB image; (3.2) The RGB image detects the instance features of people, objects, and the joint region of people and objects through the object detection model, obtains the scene category of the image through the scene classification network, and obtains the original visual features of the image through the backbone network; (3.3) Extract image-specific object availability knowledge, scene knowledge, and image caption knowledge, and embed them into the feature space; (3.4) Fuse instance features and then fuse them with availability knowledge embedding through VT-Attention to output instance-aware features; (3.5) The original visual features of the image are fused with the availability-scene knowledge embedding and the caption knowledge embedding respectively, and their channels are concatenated. After convolution to reduce the number of channels, the scene-aware features are output. (3.6) Embed instance-aware features, scene-aware features, and availability-scene knowledge into the input to predict the action category, thereby obtaining the HOI triplet, which is then combined with the detection boxes of people and objects obtained by the object detection model as the final HOI detection result.

8. A human-object interaction detection device based on availability-scene knowledge prompts, used to perform the method according to claim 1, characterized in that: The device includes: The building module, configured to construct the Availability-Scene Knowledge Graph (ASKG), first extracts the object corresponding to each image and its associated action from the dataset annotations. The images and their annotations are from the Human-Object Interaction Detection (HOI) datasets HICO-DET and V-COCO. The annotations are composed of triples.<human,action,object> The system is composed of three sets of triplets: a scene classification network is used to identify scene categories and attributes for each image, and a contextual knowledge graph is constructed.<scene,has,object> ,<object,has,affordance> and<scene,has,property> In this form, different objects, scenes, and the relationships between them are linked; the final ASKG is obtained by filtering out identical triples. The visibility-scene knowledge graph retrieves images by object and scene, generates image-related subgraphs, and extracts specific knowledge. The module is configured to build a human-object interaction detection model: During the training phase, the input is an RGB image. The first phase detects people and objects, identifies scene categories, and encodes original features. In the second phase, the prior knowledge embedding module encodes specific knowledge embeddings, the instance domain adaptive fusion module encodes instance features and knowledge embeddings to obtain instance-aware features, and the scene domain adaptive fusion module encodes scene features and knowledge embeddings to obtain scene-aware features. Finally, the HOI recognition module outputs the results. The detection module is configured to perform human-object interaction detection in images.

9. The human-object interaction detection device based on availability-scene knowledge prompts according to claim 8, characterized in that: The construction module includes the following sub-steps during the training phase: (2.1) Encode the knowledge embedding of object availability and image scene, and filter out the subgraph of all objects, object availability and scene attributes contained in the scene from the ASKG according to the scene category in the classification. The objects are filtered by comparing the detected target categories with the scene subgraph and removing knowledge that does not contain the target, thus obtaining knowledge specific to the input image, which includes: scene category, scene attributes, object category and object availability; (2.2) Extract the regions of people and objects, as well as the features of their joint spatial regions, and align them using convolutional layers respectively; then perform instance features on people. Instance characteristics of objects Human-object joint regional characteristics Summation yields the fused instance domain features. , （2）； (2.3) Embed the scene knowledge and object availability knowledge output by PKEM into With visual features Fusion yields visual availability-scene knowledge features. Simultaneously embed the coded title generated by PKEM. By introducing and fusing subtitle embedding with visual features, visual subtitle knowledge features are calculated. Visual availability-scene features and visual caption knowledge characteristics Channel concatenation is performed, and then a 1×1 convolutional layer is used to reduce the number of channels to obtain the final scene-aware features. ; (2.4) Instance-aware features Availability-Scene Knowledge Embedded Features Scene-aware features As input for human-object interaction detection, the final HOI triple is output as the action category.

Images