Method and system for character interaction detection based on cross-modal fusion of large language models

Abstract

The application discloses a character interaction detection method and system based on a large language model cross-modal fusion, and the method comprises the following steps: 1) image feature extraction; 2) multi-task feature extraction; 3) interactive behavior feature enhancement; 4) large language model text generation; and 5) cross-modal fusion. The application adopts a query-based mode to extract multiple groups of features from an image, adapts feature attention to different tasks, improves the feature representation of interactive behavior through multi-branch feature fusion and interactive behavior feature enhancement, and thus improves the accuracy and reliability of character interaction detection. Meanwhile, the powerful language expression capability of the large language model is utilized to obtain rich feature description texts, and the cross-modal fusion training is combined to align the texts and visual spaces, so that the character interaction detection has good performance in a small sample condition.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, target detection, and human interaction detection, and specifically relates to a human interaction detection method and system based on cross-modal fusion of a large language model. Background Art

[0002] The task of human interaction (HOI) detection is to locate people and objects in a given image, identify person-object pairs, and identify their interactions. The results of human interaction detection can be widely applied to computer vision problems such as video surveillance, robotics, image retrieval, and image captioning. These downstream tasks require a comprehensive visual understanding of the relationships between people and objects for high-level reasoning. Therefore, it is extremely important to develop an efficient and accurate human interaction detection method.

[0003] HOI detection methods are mainly divided into two-stage detection methods and single-stage detection methods. Two-stage detection methods mostly adopt a serial architecture. They first locate the bounding boxes of people and objects and their category labels. Then, they match people and objects one by one, generating person-object pairs that are input into a neural network to extract the features needed to predict the interaction category. These features vary in visual, spatial, pose, and semantic features. Single-stage detection methods, on the other hand, extract features from the input image, directly detecting the three-element combination of human interaction behavior, and decomposing HOI detection into multi-task learning: person-object detection and interaction behavior classification. Therefore, single-stage detection methods can easily focus on interacting person-object pairs and effectively extract corresponding features in an end-to-end approach.

[0004] Current HOI detection methods have achieved certain results. However, some problems still exist: in two-stage detection methods, far more human-object pairs are generated than the actual interactions in the image, which incurs additional computational overhead for the model and causes an imbalance in the number of positive and negative samples. In single-stage detection methods, the coupling of human-object detection and interaction behavior classification limits the performance of the model because it is difficult to generate a unified feature representation for two different tasks. At the same time, due to the complex human activities and various interaction phenomena in real scenes, HOI data has long-tail distribution and zero-sample problems, resulting in the model's lack of generalization ability for unseen interaction behaviors. Although many current methods use the CLIP model to convert HOI labels into natural language supervision to achieve zero-shot detection, a single category label may not provide rich semantic information, resulting in an inability to reveal the similarity of visual features between human-object interaction pairs.

[0005] The existing methods for detecting human interaction are not rich and comprehensive enough in expressing the characteristics of interactive behaviors, which may lead to deviations or omissions in identifying and understanding human interaction behaviors. At the same time, when introducing visual-language models to solve the long-tail and small sample problems of human interaction detection data, there is a challenge of failing to effectively align the visual-text space. Summary of the Invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and proposes a method and system for detecting human interaction based on cross-modal fusion of a large language model.

[0007] Based on the multi-task requirements of person interaction detection, the present invention extracts multi-task features to adapt to the different feature concerns of each task, and enhances the feature representation of interactive behavior through interactive behavior features. At the same time, with the help of the powerful language output capability of the large language model, text rich in feature descriptions is obtained. Combined with cross-modal fusion, the well-aligned text-visual space is effectively trained, thereby improving the classification accuracy of person interaction detection and the performance effect in the case of long-tail data distribution and few samples.

[0008] A first aspect of the present invention relates to a method for detecting human interaction based on cross-modal fusion of a large language model, comprising the following steps:

[0009] 1. Image feature extraction

[0010] First, an original image I to be detected is selected and input into a convolutional neural network to extract the image feature map. The feature map is input into a 1×1 convolution layer to reduce the channel dimension and add position encoding to the feature map. The feature map is then expanded into a feature sequence and input into the Transformer encoder to obtain the updated image feature sequence V. d .

[0011] 2. Multi-task feature extraction

[0012] Three sets of queries Q are defined for the three tasks of human-object detection and interaction behavior classification. h ,Q o ,Q ins At the same time, position-guided encoding is introduced to assign people, objects and interactive behavior queries at the same location to a group. Then, with the same image visual feature V d The positional encoding is fed into N Transformer decoder layers, which output the updated query after self-attention and cross-attention calculations. O h and O o Input into the regression classification head to predict the bounding box coordinates of people and objects and And O oInput category classification head to predict object category where K o is the number of object categories.

[0013] 3. Enhanced representation of interactive behavior features

[0014] The human visual feature O h and the visual features of the object O o Splicing to get the visual features of the person instance According to the coordinates of each group of people and objects bounding box, the spatial information of each group of people is calculated, including the center position distance, aspect ratio and intersection ratio, etc., to obtain the spatial features Then the visual features O of the person instance pair P and spatial characteristics S P Input feature fusion module MBF, output updated person instance feature Z P Then Z P Same interactive behavior characteristics ins Input decoder, calculate Z P =SelfAttn(Z P ), C ins =CrossAttn(Z P ,O ins ), and obtain the enhanced interactive behavior feature C ins . Use CLIP visual encoder to extract image features V s , C ins After the self-attention calculation is performed first, the same as V s Input the cross attention module and get the feature representation D ins .

[0015] 4. Large Language Model Text Generation

[0016] For each set of character interaction behavior labels, two sets of text prompts are designed. A large language model (such as GPT3.5) is used to generate text descriptions including the action features of the interactive human body and the feature descriptions of the objects in the interactive behavior. The CLIP text encoder is then used to encode the text to obtain text features.

[0017] 5. Cross-modal fusion

[0018] Interaction behavior features are cross-modally fused with text features. This updates the resulting text features containing visual information and visual features containing text information. This facilitates training to achieve a well-aligned visual-text space. The updated text features are then used to calculate cosine similarity with the interaction behavior features to obtain the final interaction behavior classification results.

[0019] A second aspect of the present invention relates to a person interaction detection system based on cross-modal fusion of a large language model, comprising:

[0020] Image feature extraction module, used to extract global features of the image;

[0021] The multi-task feature extraction module extracts human features, object features, and interaction features to meet the multi-task requirements of human interaction detection. It is used to predict the location of people, the location and category of objects, and the category of interaction behaviors.

[0022] Interaction behavior feature enhancement module, used to enhance interaction feature representation using person-pair instance features and spatial features;

[0023] Large language model text generation module, used to generate corresponding text descriptions based on pre-designed prompts and extract text features;

[0024] Cross-modal fusion module for bidirectionally enhancing feature representation, using text features to enhance visual features and then using the enhanced visual features to enhance text features;

[0025] The third aspect of the present invention relates to a person interaction detection device based on cross-modal fusion of a large language model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the person interaction detection method based on cross-modal fusion of a large language model of the present invention.

[0026] A fourth aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method of detecting human interaction based on cross-modal fusion of a large language model according to the present invention.

[0027] The present invention adopts a query-based approach to extract multiple sets of features from images to adapt to the feature focus of different tasks, and improves the feature representation of interactive behavior through multi-branch feature fusion and interactive behavior feature enhancement, thereby improving the accuracy and reliability of human interaction detection; at the same time, it utilizes the powerful language expression ability of the large language model to obtain rich feature description text, and combines cross-modal fusion training with well-aligned text-visual space to achieve good performance of human interaction detection in the case of few samples.

[0028] The advantages of the present invention are: according to the different features focused on by different tasks of human interaction detection, a query-based method is used to extract multiple sets of features from the image, and the feature expression of interactive behavior is improved through multi-branch feature fusion and interactive behavior feature enhancement, thereby improving the accuracy and reliability of human interaction detection; at the same time, a large language model and cross-modal fusion are used to obtain richer descriptive text information, effectively train a well-aligned text-visual space, and achieve good performance of human interaction detection with a small number of samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a network structure diagram of character interaction detection based on cross-modal fusion of a large language model in the present invention.

[0030] Figure 2 It is a schematic diagram of the multi-branch feature fusion and interaction feature enhancement module of the present invention for character interaction detection based on cross-modal fusion of a large language model.

[0031] Figure 3 It is a schematic diagram of a cross-modal fusion module for character interaction detection based on cross-modal fusion of a large language model of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is explained and described clearly and completely below in conjunction with the accompanying drawings.

[0033] Example 1

[0034] Reference Figures 1 to 3 This embodiment relates to a method for detecting human interaction based on cross-modal fusion of a large language model, comprising the following steps:

[0035] 1. Image feature extraction

[0036] The original image I to be detected is input into the convolutional neural network to extract the image feature map. The feature map is input into a 1×1 convolution layer to reduce the channel dimension and add position encoding to the feature map. The feature map is then expanded into a feature sequence and input into the Transformer encoder to obtain the updated image feature sequence V d .

[0037] 2. Multi-task feature extraction

[0038] Three sets of queries Q are defined for the three tasks of human-object detection and interaction behavior classification. h ,Q o ,Q ins At the same time, position-guided encoding is introduced to assign people, objects and interactive behavior queries at the same location to a group. Then, with the same image visual feature V dThe positional encoding is fed into N Transformer decoder layers, which output the updated query after self-attention and cross-attention calculations. Q h and O o Input into the regression classification head to predict the bounding box coordinates of people and objects and And O o Input category classification head to predict object category where K o is the number of object categories.

[0039] 3. Enhanced representation of interactive behavior features

[0040] The human visual feature O h and the visual features of the object O o Splicing to get the visual features of the person instance According to the coordinates of each group of people and objects bounding box, the spatial information of each group of people is calculated, including the center position distance, aspect ratio and intersection ratio, etc., to obtain the spatial features Then the visual features O of the person instance pair P and spatial characteristics S P Input feature fusion module MBF, output updated person instance feature Z P Then Z P Same interactive behavior characteristics ins Input decoder, calculate Z P =SelfAttn(Z P ), C ins =CrossAttn(Z P ,O ins ), and obtain the enhanced interactive behavior feature C ins . Use CLIP visual encoder to extract image features V s , C ins After the self-attention calculation is performed first, the same as V s Input the cross attention module and get the feature representation D ins .

[0041] 4. Large Language Model Text Generation

[0042] For each interactive behavior label, two sets of prompt inputs are designed into a large language model (such as GPT3.5). The large language model outputs the subject action feature description and object feature description corresponding to each interactive behavior, and uses the CLIP text encoder to encode the generated text to obtain the text feature T.

[0043] 5. Cross-modal fusion

[0044] Denote the interactive behavior feature in step 3 as D ins Perform cross-modal fusion on the text feature T in step 4. Update the text features containing visual information and the visual features containing text information. Then, use the updated text features to calculate the cosine similarity with the interaction behavior features to obtain the final interaction behavior classification result.

[0045] The method of the present invention overcomes the problems of current human interaction detection methods, such as the lack of rich and comprehensive representation of human interaction features and poor alignment of text and visual space. It improves the accuracy of human interaction detection and its performance in the case of few samples.

[0046] Example 2

[0047] This embodiment relates to a person interaction detection system based on large language model cross-modal fusion, which is used to implement the person interaction detection method based on large language model cross-modal fusion of Example 1, including:

[0048] Image feature extraction module, used to extract global features of the image;

[0049] The multi-task feature extraction module extracts human features, object features, and interaction features to meet the multi-task requirements of human interaction detection. It is used to predict the location of people, the location and category of objects, and the category of interaction behaviors.

[0050] Interaction feature enhancement module, used to enhance the interaction feature representation using the person-pair instance visual features and spatial features;

[0051] Large language model text generation module, used to generate corresponding text descriptions based on pre-designed prompts and extract text features;

[0052] Cross-modal fusion module for bidirectionally enhancing feature representation, using text features to enhance visual features and then using the enhanced visual features to enhance text features;

[0053] Example 3

[0054] This embodiment relates to a person interaction detection device based on cross-modal fusion of a large language model, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the person interaction detection method based on cross-modal fusion of a large language model in Example 1.

[0055] Example 4

[0056] This embodiment relates to a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for detecting human interaction based on cross-modal fusion of a large language model in embodiment 1 is implemented.

[0057] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for detecting person interactions based on cross-modal fusion of a large language model, including the following steps: Step 1: Image feature extraction: This method combines the deep representation capabilities of convolutional neural networks with the global context understanding capabilities of Transformer encoders to extract global features of the image. Step 2: Multi-task feature extraction: Based on the query vector, the Transformer decoder extracts human features, object features, and interaction behavior features from the image. The extracted human and object features are then fed into a multi-layer perceptron to predict the location and category of the person and object, respectively. Step 3: Enhance the interactive behavior feature representation. Use a multi-branch fusion module to fuse the person-pair instance features and spatial features, and use the fused features to enhance the interactive behavior feature representation. Step 4. Large language model text generation: Define text prompts and use the large language model to generate descriptive text for each pair of person interactions, including text describing the action features of the people in the interaction and the features of the objects in the interaction. Step 5. Cross-modal fusion: Use text features to enhance visual features, and use the enhanced visual features to enhance text features, and train a well-aligned text-visual space.

2. The method for detecting person interactions based on large language model cross-modal fusion according to claim 1, characterized in that: Step 1 includes: First, select an original image to be detected I , input into the convolutional neural network to extract the image feature map; input the feature map into The convolutional layer reduces the channel dimension and adds position encoding to the feature map; then the feature map is expanded into a feature sequence and input into the Transformer encoder to obtain the updated image feature sequence .

3. The method for detecting person interactions based on large language model cross-modal fusion according to claim 1, characterized in that: Step 2 includes: Define query vectors for the three tasks of human-object detection and interaction behavior classification and , guiding the Transformer decoder to extract human features from the image , object features and interactive behavior characteristics ; Then the extracted human features and object features are input into the multi-layer perceptron to predict the position of the person respectively , the position of the object and object categories ,in is the number of object categories.

4. The method for detecting person interactions based on large language model cross-modal fusion according to claim 1, characterized in that: Step 3 includes: The human features extracted in step 2 and object features Splicing to get the character instance features ; Construct the spatial features of each group of people based on the predicted coordinate frame information Then, the two sets of features are input into the multi-branch fusion module to fuse the character pair instance features and spatial features to generate more expressive and discriminative character pair features. ; then Same interactive behavior characteristics as in step 2 Input the interactive feature enhancement module, and after self-attention calculation and cross-attention calculation, output the interactive behavior feature representation rich in contextual relationship and spatial location information ; Finally, and image features extracted using CLIP visual encoder Input the character interaction decoder together and output the final interaction features .

5. The method for detecting human interaction based on large language model cross-modal fusion according to claim 1, characterized in that: Step 4 includes: For each pair of character interaction behavior labels, two sets of text prompts are defined. The large language model is used to generate the description text of each pair of character interaction behaviors, including the action feature description text of the people in the interaction pair and the feature description text of the objects in the interaction pair. Then, the CLIP text encoder is used to encode the description text of each pair of character interaction behaviors to obtain text features. .

6. The method for detecting person interactions based on large language model cross-modal fusion according to claim 1, characterized in that: Step 5 includes: The text features encoded in step 4 Same visual features as in step 3 Perform cross-modal fusion; use text features to enhance visual features, and use the enhanced visual features to enhance text features; then calculate the cosine similarity between the updated visual features and the updated text features to predict the interaction category of each group of person interaction pairs.

7. A cross-modal character interaction detection system based on a large language model, characterized by: include: Image feature extraction module, used to extract global features of the image; The multi-task feature extraction module extracts human features, object features, and interaction features to meet the multi-task requirements of human interaction detection. It is used to predict the location of people, the location and category of objects, and the category of interaction behaviors. Interaction behavior feature enhancement module, used to enhance interaction feature representation using person-pair instance features and spatial features; Large language model text generation module, used to generate corresponding text descriptions based on pre-designed prompts and extract text features; A cross-modal fusion module is used to enhance feature representation bidirectionally, using text features to enhance visual features and then using the enhanced visual features to enhance text features.

8. A person interaction detection device based on cross-modal fusion of a large language model, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for detecting human interaction based on cross-modal fusion of a large language model according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by the processor, the method for detecting human interaction based on cross-modal fusion of a large language model according to any one of claims 1 to 6 is implemented.

Images