An open-vocabulary human-robot interaction detection method based on a calibrated diffusion model

Abstract

The application discloses an open-vocabulary human-object interaction detection method based on a calibrated diffusion model and relates to the technical fields of computer vision and artificial intelligence. The method adopts a two-stage teacher-student learning framework: a teacher model calibrates a generative diffusion model by injecting human-computer interaction prior knowledge to generate low-noise and fine-grained visual features for human-object interaction regions; a student model is an end-to-end detection framework that is optimized as a whole under the supervision of a hybrid loss, thereby avoiding suboptimal problems caused by multi-stage independent training, making the detection process more efficient and the performance more optimal. By introducing a human-object pair learner and a local perception decoder and combining knowledge distillation of the teacher model, the method can more deeply understand the essential semantics and spatial relationships of interaction and has stronger generalization ability for novel and unseen interaction scenes. The method can effectively suppress the internal noise of a pre-training model and make the features highly aligned with an HOI task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, and particularly relates to an open-vocabulary human-object interaction detection method based on a calibrated diffusion model. BACKGROUND

[0002] In application scenarios such as intelligent monitoring, human-computer interaction, and autonomous driving, it is crucial to accurately recognize and understand the interaction relationship between people and objects in images, which is referred to as human-object interaction (HOI) detection in academia. Existing HOI detection techniques are mostly studied under the "closed set" setting, that is, all interaction categories to be detected are known during model training. The limitation of this method is that there are numerous and emerging interaction categories in the real world, and once the model encounters a novel interaction not included in the training data, it cannot effectively identify it, which greatly reduces its application value in real complex environments.

[0003] To solve this problem, researchers have begun to explore "open-vocabulary" HOI detection, aiming to enable the model to have generalization ability to recognize unseen interactions. Recent solutions attempt to use large-scale visual language pre-training models (VLM) to leverage their rich world knowledge. However, when these VLMs are directly applied to the HOI detection task, a series of technical bottlenecks are often encountered. For example, these VLMs are usually trained using massive amounts of image-text data collected from the Internet, which inherently contains a lot of noise, and are more focused on object recognition rather than interaction between objects, which leads to a deviation between the visual features extracted by VLM and the fine HOI interaction semantics, resulting in poor alignment.

[0004] VLM aims to learn the global association between the entire image and the corresponding text, and the features generated by VLM are too coarse for the HOI task, which requires accurate capture of fine spatial and semantic relationships between people and objects in local regions, and cannot provide effective local detail information. At the same time, this also leads to the model "remembering" the common human-object spatial layout in the dataset during training, and when encountering an uncommon spatial configuration during testing, the model performance will significantly decrease, and the generalization ability is insufficient. Therefore, there are still challenges in realizing efficient and accurate open-vocabulary HOI detection in the field, and it is necessary to develop a technical solution that can effectively suppress feature noise, enhance local detail perception, and robustly generalize to novel interactions. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an open-vocabulary human-object interaction detection method based on a calibrated diffusion model, to solve the problem of poor detection performance on unseen interaction categories caused by the internal noise of visual language models (VLM), global feature deviation, and spatial relationship overfitting in the prior art.

[0006] The present application solves the above technical problems by adopting the technical solution of:

[0007] The present application adopts an innovative "teacher-student" two-stage training framework, and solves the above technical problems by adopting the technical solution of:

[0008] An open vocabulary human-machine interaction detection method based on a calibrated diffusion model, comprising the following steps:

[0009] First stage: calibration and training of the teacher model;

[0010] Step 1: Obtain an image dataset with "human-object interaction" labels, crop the minimum bounding rectangle of each interaction pair as an image sample, and generate a corresponding "human-action-object" format text description;

[0011] Step 2: Align the visual features of the image sample with the semantic features of the corresponding text description to obtain a knowledge base containing various interaction modes;

[0012] Second stage: training of the student model HOI detector;

[0013] Step 3: Use a pre-trained object detector to process the complete image to generate candidate human and object regions and their features, and fuse their spatial position information;

[0014] Step 4: Use the features obtained in step 3 to construct a graph structure, then use a graph network learning module to perform relationship reasoning and filtering on the candidate human-object pairs, and generate optimized human-object pair fusion features;

[0015] Step 5: Use an interaction decoder to decode the fusion features, and use a hybrid loss function to supervise and optimize the student model end-to-end, introduce the soft labels generated by the teacher model, and transfer the generalization knowledge of the teacher model to the student model;

[0016] Third stage: prediction process;

[0017] Step 6: Use the trained HOI detector to perform subsequent detection tasks.

[0018] Further, the step 1 is specifically as follows:

[0019] Step 1.1: Obtain sample picture data and interaction labels;

[0020] Step 1.2: Extract the joint region of the human-object pair as a visual sample, calculate the minimum bounding rectangle for each labeled human and object bounding box, and then crop the corresponding image region from the original image;

[0021] Step 1.3: Build a text input sample, and use a preset text template to generate corresponding text descriptions for all human-computer interaction (HOI) categories.

[0022] Further, the step 2 adopts a visual and text dual-branch structure for feature calibration, specifically as follows:

[0023] Step 2.1: Build a visual coding branch to calibrate the extraction of visual features; the visual coding branch includes a frozen contrastive language-image pre-training (CLIP) image encoder, a learnable adapter, a pre-trained diffusion model U-Net network, and a feature fusion encoder.

[0024] First, input the image region into the frozen CLIP image encoder and the learnable adapter to map it into an implicit text embedding.

[0025] Input the image region and the implicit text embedding into the diffusion model U-Net network together, and through the cross-attention mechanism inside the U-Net network, fuse the image features and the text embedding to generate multi-scale feature maps aligned with the HOI semantics.

[0026] Input the multi-scale feature maps into the feature fusion encoder for processing; specifically, first reduce the channel dimension of each scale of feature map, then unify all feature maps to the same spatial size through upsampling and concatenate along the channel axis, finally refine through residual connection and batch normalization, and apply global average pooling to generate a calibrated visual embedding vector.

[0027] Step 2.2: Build a text coding branch, which includes a pre-trained CLIP text encoder and a learnable adapter; encode the text description to finally generate an HOI knowledge base aligned with the HOI task.

[0028] Step 2.3: Alignment and optimization: match the calibrated visual embedding vector with the HOI knowledge base, and calculate the similarity score of each HOI category.

[0029] Step 2.4: Use the binary cross-entropy loss function as the supervision signal to optimize the learnable parameters in the teacher model.

[0030] Further, the step 3 is specifically as follows:

[0031] Step 3.1: Use a pre-trained object detector to process the input image and generate candidate region proposals and their preliminary features for people and objects.

[0032] Step 3.2: First, extract the bounding box coordinates of each candidate region and perform sinusoidal position encoding to obtain spatial prior information, then input the spatial prior information and the preliminary features into the self-attention module for fusion to obtain spatial perception features.

[0033] Further, the step 4 is specifically as follows:

[0034] Step 4.1: based on the similarity between each feature vector in the spatial perception feature, a graph structure G is constructed, and then a graph convolution operation is performed on the graph structure G to suppress unreasonable interaction combinations, to obtain updated features;

[0035] Step 4.2: the features of the person and the object with the highest score in the updated features are spliced, and are fused through a multi-layer perception machine MLP to obtain a final interaction query as a person-object pair fusion feature.

[0036] Further, the interaction decoder comprises 2 decoding layers, each decoding layer comprising a multi-head self-attention module and a multi-head cross-attention module, and the multi-head attention mechanism is set to 8 heads; at each layer of the decoder, the pair-wise spatial prior information of the person-object pair is explicitly injected into the multi-head self-attention and multi-head cross-attention modules.

[0037] Further, the hybrid loss function is represented by the following formula:

[0038] ;

[0039] wherein, is a pair loss for supervising the person-object pair learner; is a hard label loss calculated by using a real label; is a soft label loss; is a balance hyperparameter.

[0040] The soft label loss is realized through a knowledge distillation mechanism; in order to avoid the conflict between the soft label and the hard label, a mask is introduced, which ensures that the soft label only takes effect on the categories that are not covered by the real label, and guides the student model to learn unseen interactions, and the mask is defined by the following formula:

[0041] ;

[0042] wherein, is a soft label generated by a teacher model, is a real label.

[0043] Further, the learnable adapter is specifically a four-layer multi-layer perception machine.

[0044] The beneficial effects of the present application are:

[0045] (1) Significantly improve the detection accuracy of open vocabulary: the method proposed in the present application can effectively suppress the internal noise of the pre-trained model, and make the features highly aligned with the HOI task, thereby achieving a significant improvement of about 3 percentage points in detection accuracy (mAP) for unseen interaction categories on multiple authoritative public datasets compared with the existing optimal technology.

[0046] (2) Enhance model generalization and robustness: by introducing a person-object pair learner and a local perception decoder, and combining the knowledge distillation (soft label guidance) of the teacher model, the method can more deeply understand the essential semantics and spatial relationships of interaction, rather than just memorizing the training data distribution, thus having stronger generalization ability for novel and unseen interaction scenarios.

[0047] (3) End-to-end optimization is realized: the student model of the present application is an end-to-end detection framework that is optimized as a whole under the supervision of a hybrid loss, avoiding suboptimal problems caused by multi-stage independent training, making the detection process more efficient and the performance better. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the overall flowchart of the "teacher-student" framework in the embodiment of the present application;

[0049] Figure 2 is the system architecture diagram of the training and calibration process of the "teacher model" in the embodiment of the present application;

[0050] Figure 3 is the system architecture diagram of the training process of the "student model" in the embodiment of the present application. DETAILED DESCRIPTION

[0051] The present application aims to propose an open vocabulary human-computer interaction detection method based on a calibrated diffusion model to solve the problem of poor detection performance of unseen interactions caused by relying on noisy visual language models in the prior art. The method includes a training process and a prediction process, as shown in Figure 1 In the training process, a two-stage teacher-student learning framework is adopted. First, a calibrated diffusion model is trained as a teacher to generate low-noise visual features. Then, a HOI detector is trained as a student using the teacher model for guidance. The detector internally includes a person-object pair learner and a local perception decoder, and is jointly supervised using knowledge distillation. In the prediction process, the trained student model is used for end-to-end detection of images.

[0052] EMBODIMENT:

[0053] The open vocabulary human-computer interaction detection method based on the calibrated diffusion model in the present embodiment includes a training process and a prediction process, which will be described in detail below with reference to the accompanying drawings.

[0054] Referring to Figure 2 and Figure 3 , the training process mainly includes two stages:

[0055] The first stage: calibration and training of the teacher model CaDiff.

[0056] Step 1, obtain an image dataset with "human-object interaction" (HOI) annotation, which can use public datasets such as HICO-DET and V-COCO; crop the minimum bounding rectangle for each interaction pair as an image sample, and generate the corresponding "person-action-object" format text description;

[0057] Step 1.1, obtain sample picture data and interaction label;

[0058] Step 1.2, extract the joint region of human-object pairs as visual samples. For each annotated human and object bounding box and , calculate its minimum bounding rectangle, and then crop the corresponding image region from the original image . This process can be represented by the following formula:

[0059]

[0060] where, is the original input image; and are the bounding boxes of the person and object, respectively; is a function that calculates the minimum bounding rectangle; is a function that extracts the corresponding image region according to the bounding rectangle; is the final cropped image region.

[0061] Step 1.3, construct the text input sample, use the preset text template "a photo of a person{verb} {object}", generate the corresponding text description for all HOI categories.

[0062] Step 2, use a visual and text dual-branch structure for feature calibration; align the visual features of the image sample with the semantic features of the corresponding text description in depth, thereby constructing a knowledge base containing various interaction modes;

[0063] Step 2.1, the extraction of visual features is as follows:

[0064] Step 2.1.1. Building the visual encoding branch; this branch consists of a frozen contrastive language-image pre-training (CLIP) image encoder, a learnable adapter, a pre-trained diffusion model U-Net network, and a feature fusion encoder. In this embodiment, the adapter is specifically a four-layer multi-layer perceptron (MLP).

[0065] Step 2.1.2. Generating implicit text conditioning:

[0066] The image region processed in Step 1.2 is input into the frozen CLIP image encoder and the learnable adapter to map its visual features into an implicit text embedding . This process can be represented by the following equation:

[0067]

[0068] where is the frozen CLIP image encoder; is the learnable adapter; is the generated implicit text embedding.

[0069] Step 2.1.3. Extracting features with the diffusion model: the image region and the implicit text embedding obtained in Step 2.1.2 are jointly input into the U-Net network of the diffusion model. Through the cross-attention mechanism inside the U-Net network, the image features and the text embedding are fused, generating a multi-scale feature map that guides the network to generate HOI semantics-aligned features. This process can be represented by the following equation:

[0070]

[0071] where is the generated multi-scale feature map, is the U-Net network of the diffusion model.

[0072] Step 2.1.4. Feature fusion: the multi-scale feature map output by the U-Net is input into the fusion encoder for processing. Specifically, first, a 1x1 convolution is used to reduce the channel dimension of each scale of the feature map, then all the feature maps are unified to the same spatial size through upsampling and spliced along the channel axis, and finally a module containing residual connection and batch normalization is used for refinement, and a global average pooling ( ) is applied to generate the final, compact calibration visual embedding vector .

[0073] Step 2.2, the extraction of HOI text features, is as follows:

[0074] Step 2.2.1, a text encoding branch is built. This branch consists of a pre-trained CLIP text encoder and a learnable adapter, which is also a four-layer MLP.

[0075] Step 2.2.2, the HOI class text descriptions generated in Step 1.3 are encoded through the text encoding branch, finally generating an optimized HOI knowledge base aligned with the HOI task .

[0076] Step 3, model alignment and optimization:

[0077] Step 3.1, for each region-level visual embedding vector extracted in the previous step, match it with the HOI knowledge base generated in Step 2.2.2 to calculate the similarity score of the region belonging to each HOI class :

[0078]

[0079] where, is the cosine similarity function; the HOI knowledge base is a set containing class text embedding vectors , is the element representing the th HOI class in this set.

[0080] Step 3.2, use the binary cross-entropy (BCE) loss function as the supervision signal to optimize the learnable parameters in the teacher model. In this embodiment, the Adam optimizer is used, with a learning rate of 5e-5, and the pre-training process is performed for 40 epochs.

[0081] Second stage: training of the student model (HOI detector).

[0082] Step 1, candidate generation and spatial encoding; use a pre-trained object detector to process the complete image to generate candidate human and object regions and their features, and fuse their spatial position information.

[0083] Step 1.1, use a pre-trained object detector whose backbone network can be ResNet-50 or Swin-L to process the input image to generate candidate region proposals and preliminary features of people and objects.

[0084] ​Step 1.2, spatial-aware encoding on the candidate region proposals generated in S11. Specifically, first, the bounding box coordinates of each candidate region are extracted and are encoded by a sinusoidal position encoding to obtain the spatial prior information. Then, the spatial prior information is input into a self-attention module together with the preliminary features (also referred to as candidate features herein) output by S11 for fusion, and finally, a feature representation rich in spatial relationships is obtained as the spatial-aware feature .

[0085] Step 2, human-object pair learning and optimization: a graph network learning module is used to perform relationship reasoning and screening on the candidate human-object pairs, to suppress unreasonable combinations, and to generate a fused feature of the optimized human-object pairs;

[0086] Step 2.1.1, a lightweight human-object pair learner based on a graph convolutional network (GCN) is built to filter and optimize the “human-object” candidate combinations. The learner first constructs a graph structure G according to the similarity between the feature vectors in the spatial-aware feature H' obtained in step 1.2; then, graph convolution operations are performed on the graph structure G to suppress unreasonable interaction combinations. This process can be represented by the following formula:

[0087]

[0088] wherein, is the input spatial-aware feature; is the adjacency matrix of the graph, and the similarity threshold t used for construction is set to 0.6; is a learnable parameter of ; is the updated feature.

[0089] Step 2.1.2, the features of the “human-object” pairs with the highest scores that have been filtered and optimized by the GCN and are spliced, and a multi-layer perceptron is used to fuse them to obtain the final interaction query .

[0090]

[0091] wherein, is a splicing operation.

[0092] Step 3, local-aware decoding and joint supervision: an interaction decoder is used to decode the fused feature. A hybrid loss function is used to supervise and optimize the student model end-to-end, and the function not only uses real “hard labels”, but also introduces “soft labels” generated by a teacher model through a knowledge distillation mechanism, so as to transfer the generalization knowledge of the teacher model to the student model.​

[0093] Step 3.1.1, decode the interaction query generated in step 2.1.2 using a local-aware interaction decoder. In this embodiment, the decoder contains 2 decoding layers, each of which includes a multi-head self-attention module and a multi-head cross-attention module, and the multi-head attention mechanism is set to 8 heads.

[0094] Step 3.1.2, explicitly inject the pair-wise spatial prior information of human-object pairs into the multi-head self-attention and multi-head cross-attention modules at each layer of the decoder. This is done to enable the decoder to generate more discriminative feature representations that incorporate spatial location relationships, providing a solid foundation for the final accurate prediction of interaction actions.

[0095] Step 3.2.1, use a hybrid loss function consisting of three parts to perform end-to-end joint supervision optimization of the student model. The total loss function is represented by the following formula:

[0096]

[0097] where, is the pair loss for supervising the human-object pair learner; is the hard label loss calculated using the real label; is the soft label loss; is the balance hyperparameter, which is set to 350 in this embodiment.

[0098] Step 3.2.2, the soft label loss is achieved through a knowledge distillation mechanism. To avoid conflicts between soft labels and hard labels, a mask is introduced. This mask ensures that soft labels only take effect for classes that are not covered by the real label, guiding the student model to learn unseen interactions. The mask is defined by the following formula:

[0099]

[0100] where, is the soft label generated by the teacher model, is the real label.

[0101] The prediction process flow in this embodiment is described in Figure 3 , which includes:

[0102] Step 1, data input:

[0103] Step 1.1, directly input an image to be predicted into the student HOI detector trained in the second stage.

[0104] Step 2, end-to-end detection: ​

[0105] Step 2.1, the student model automatically performs a complete forward propagation process, including: target candidate generation, human-object pair optimization, and interaction decoding, outputting the interaction verb prediction score of each valid "human-object" pair;

[0106] Step 3, score fusion and output:

[0107] Step 3.1, the interaction verb prediction score obtained in step 2.1 is fused with the detection confidence score of the corresponding human ( ) and object ( ), to calculate the final HOI confidence score . The fusion process can be represented by the following formula:

[0108]

[0109] wherein, is a hyperparameter for balancing each score, which is set to 0.26 in the present embodiment.

[0110] Step 3.2, according to the final confidence score , all detected HOI triplets are sorted, and the result with the highest score is output.

[0111] It can be understood that the present application is described through some embodiments, and those skilled in the art know that various changes or equivalent replacements can be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments can be modified to adapt to specific conditions and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application are within the scope of the present application.​

Claims

1. An open-vocabulary human-robot interaction detection method based on a calibrated diffusion model, characterized in that, Comprising the following steps: First stage: calibration and training of the teacher model; Step 1: Obtain an image dataset with "human-object interaction" annotations, crop the minimum bounding rectangle for each interaction pair as an image sample, and generate a corresponding text description in the "human-action-object" format; Step 2: Align the visual features of the image sample with the semantic features of the corresponding text description to obtain a knowledge base containing various interaction modes; Step 2 uses a visual and text dual-branch structure for feature calibration, as follows: Step 2.1: Build a visual encoding branch to calibrate the extraction of visual features; the visual encoding branch includes a frozen contrastive language-image pre-training CLIP image encoder, a learnable adapter, a pre-trained diffusion model U-Net network, and a feature fusion encoder; First, input the image region into the frozen CLIP image encoder and the learnable adapter to map it to an implicit text embedding; Input the image region and the implicit text embedding into the diffusion model U-Net network, which fuses image features and text embeddings through the cross-attention mechanism within the U-Net network to generate multi-scale feature maps aligned with HOI semantics; The multi-scale feature maps are input into the feature fusion encoder for processing; specifically, first reduce the channel dimension of each scale's feature map, then unify all feature maps to the same spatial size through upsampling and concatenate them along the channel axis, finally refine them through residual connection and batch normalization, and apply global average pooling to generate calibrated visual embedding vectors; Step 2.2: Build a text encoding branch, which includes a pre-trained CLIP text encoder and a learnable adapter; encode the text description to generate an HOI knowledge base aligned with the HOI task; Step 2.3: Alignment and optimization: match the calibrated visual embedding vectors with the HOI knowledge base and calculate the similarity scores for each HOI class; Step 2.4: Use a binary cross-entropy loss function as a supervision signal to optimize the learnable parameters in the teacher model; Second stage: training of the student model HOI detector; Step 3: Use a pre-trained object detector to process the complete image, generating candidate human and object regions and their features, and fusing their spatial location information; Step 4: Use the features obtained in Step 3 to construct a graph structure, then use a graph network learning module to reason and filter the candidate human-object pairs, and generate optimized human-object pair fusion features; Step 5: Use an interaction decoder to decode the fusion features, and use a hybrid loss function to supervise and optimize the student model end-to-end, introducing the soft labels generated by the teacher model to transfer the generalization knowledge of the teacher model to the student model; the hybrid loss function is represented by the following formula: ； wherein, is a pair loss for supervising the human-object pair learner; is a hard label loss calculated using the true label; is a soft label loss; is a balancing hyperparameter; Third stage: prediction process; Step 6: Use the trained HOI detector for subsequent detection tasks.

2. The open-vocabulary human-object interaction detection method based on a calibrated diffusion model according to claim 1, wherein, Step 1 is as follows: Step 1.1: Obtain sample picture data and interaction labels; Step 1.2: Extract the joint region of the human-object pair as the visual sample, and calculate the minimum circumscribed rectangle for the boundary box of each annotated human-object pair, then crop the corresponding image region from the original image; Step 1.3: Build a text input sample, and generate the corresponding text description for all human-object interaction (HOI) categories using a preset text template.

3. The open-vocabulary human-object interaction detection method based on a calibrated diffusion model according to claim 2, characterized in that, The step 3 is specifically as follows: Step 3.1: A pre-trained object detector is used to process the input image to generate candidate regions of people and objects and their preliminary features; Step 3.2: First, the boundary box coordinates of each candidate region are extracted and subjected to sinusoidal position encoding to obtain spatial prior information, and then the spatial prior information is input into the self-attention module together with the preliminary features for fusion to obtain spatial perception features.

4. The open-vocabulary human-object interaction detection method based on a calibrated diffusion model according to claim 3, wherein, The step 4 is specifically as follows: Step 4.1: Based on the similarity between each feature vector in the spatial perception feature, a graph structure G is constructed, and then graph convolution operation is performed on the graph structure G to suppress unreasonable interaction combinations to obtain updated features; Step 4.2: The features of the highest scoring person and object in the updated features are spliced and fused through a multi-layer perception (MLP) to obtain the final interaction query as the human-object pair fusion feature.

5. The open-vocabulary human-object interaction detection method based on a calibrated diffusion model according to claim 4, wherein, The interaction decoder includes two decoding layers, each decoding layer includes a multi-head self-attention module and a multi-head cross-attention module, and the multi-head attention mechanism is set to 8 heads; In each layer of the decoder, the pair-wise spatial prior information of the human-object pair is explicitly injected into the multi-head self-attention and multi-head cross-attention modules.

6. The open-vocabulary human-object interaction detection method based on a calibrated diffusion model according to claim 5, wherein, The soft label loss is realized through a knowledge distillation mechanism; to avoid conflict between the soft label and the hard label, a mask is introduced The mask ensures that the soft label only takes effect for the classes that are not covered by the real label, guiding the student model to learn the unseen interactions, and the mask is defined by the following formula: ； wherein, soft labels generated for the teacher model, are true labels.

7. The open-vocabulary human-object interaction detection method based on a calibrated diffusion model according to claim 6, wherein, The learnable adapter is specifically a four-layer multi-layer perception.

Images