Character interaction detection method and system based on large language model cross-modal fusion

Abstract

The invention discloses a character interaction detection method and system based on large language model cross-modal fusion. The method comprises the following steps: 1) extracting image features; 2) multi-task feature extraction; 3) interaction behavior feature enhancement; 4) generating a large language model text; and 5) performing cross-modal fusion. According to the method, multiple groups of features are extracted from the image based on query to adapt to feature attention of different tasks, and feature representation of interaction behaviors is improved through multi-branch feature fusion and interaction behavior feature enhancement, so that the accuracy and reliability of character interaction detection are improved; and meanwhile, rich feature description texts are obtained by utilizing the powerful language expression capability of a large language model, and a good expression effect of character interaction detection under the condition of few samples is realized by combining cross-modal fusion training and aligning a good text-visual space.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision, target detection, and human interaction detection, and specifically relates to a human interaction detection method and system based on cross-modal fusion of a large language model. Background Art

[0002] The task of human interaction (HOI) detection is to locate people and objects from a given image, determine the pairing of people and objects, and recognize the interaction behaviors between them. The results of human interaction detection can be widely used in computer vision problems such as video surveillance, robotics, image retrieval, and image description. These downstream tasks require a comprehensive visual understanding of the relationship between people and objects for high-level reasoning. Therefore, it is very important to implement an efficient and accurate human interaction detection method.

[0003] HOI detection methods are mainly divided into two-stage detection methods and single-stage detection methods. Most two-stage detection methods adopt a serial architecture, first locating the bounding boxes of people and objects and their category labels, then matching people and objects one by one, generating person-object pairs and inputting them into the neural network to extract the features required to predict the interaction category. These features differ in vision, space, posture, and semantics. The single-stage detection method extracts features from the input image, directly detects the three-element combination of human interaction behavior, and decomposes HOI detection into multi-task learning, namely person-object detection and interaction behavior classification. Therefore, the single-stage detection method can easily focus on the interactive person-object pairs and effectively extract the corresponding features in an end-to-end method.

[0004] Current HOI detection methods have achieved certain results. However, there are still some problems: in the two-stage detection method, far more human-object pairs are generated than the actual interactions in the image, which causes additional computational overhead for the model and an imbalance in the number of positive and negative samples. In the single-stage detection method, the coupling of human-object detection and interaction behavior classification limits the performance of the model because it is difficult to generate a unified feature representation for two different tasks. At the same time, due to the complex human activities and various interaction phenomena in real scenes, HOI data has the problems of long-tail distribution and zero samples, resulting in the lack of generalization ability of the model for unseen interaction behaviors. Although many current methods use the CLIP model to convert HOI labels into natural language supervision to achieve zero-sample detection, a single category label may not provide rich semantic information, resulting in its inability to reveal the similarity of visual features between human-object interaction pairs.

[0005] The existing methods for detecting human interaction are not rich and comprehensive enough in expressing the characteristics of interactive behaviors, which may lead to deviations or omissions in identifying and understanding human interaction behaviors. At the same time, when introducing visual-language models to solve the long-tail and small sample problems of human interaction detection data, there is a challenge of failing to effectively align the visual-text space. Summary of the invention

[0006] The present invention aims to overcome the above-mentioned shortcomings of the prior art and proposes a method and system for detecting human interaction based on cross-modal fusion of a large language model.

[0007] According to the multi-task requirements of character interaction detection, the present invention extracts multi-task features to adapt to the different feature concerns of each task, and enhances the feature representation of interactive behavior through interactive behavior feature enhancement. At the same time, with the help of the powerful language output capability of the large language model, text rich in feature description is obtained, and combined with cross-modal fusion, the well-aligned text-visual space is effectively trained to improve the classification accuracy of character interaction detection, as well as the performance effect in the case of long-tail distribution of data and few samples.

[0008] A first aspect of the present invention relates to a method for detecting human interaction based on cross-modal fusion of a large language model, comprising the following steps:

[0009] 1. Image feature extraction

[0010] First, select an original image I to be detected and input it into the convolutional neural network to extract the image feature map. The feature map is input into a 1×1 convolution layer to reduce the channel dimension and add position encoding to the feature map. The feature map is then expanded into a feature sequence and input into the Transformer encoder to obtain the updated image feature sequence V d .

[0011] 2. Multi-task feature extraction

[0012] Three sets of queries Q are defined for the three tasks of human-object detection and interaction behavior classification. h ,Q o ,Q ins At the same time, position-guided encoding is introduced to assign people, objects, and interactive behavior queries at the same location to a group. Then, with the same image visual feature V d The positional encoding is fed into N Transformer decoder layers, which output the updated query after self-attention and cross-attention calculations. O h and O o Input to the regression classification head to predict the bounding box coordinates of people and objects and And O oInput category classification head to predict object category Where K o is the number of object categories.

[0013] 3. Enhanced representation of interactive behavior features

[0014] The visual features of human beings h and the visual features of the object O o Splicing to get the visual features of the person instance According to the coordinates of each group of people and object bounding boxes, the spatial information of each group of people is calculated, including the center position distance, aspect ratio, intersection ratio, etc., to obtain the spatial features Then the visual features O of the person instance pair are P and spatial characteristics S P Input feature fusion module MBF, output updated person instance feature Z P Then Z P Interaction behavior characteristics ins Input decoder, calculate Z P =SelfAttn(Z P ), C ins =CrossAttn(Z P ,O ins ), and obtain the enhanced interactive behavior feature C ins . Use CLIP visual encoder to extract image features V s , C ins After the self-attention calculation is performed, the same as V s Input into the cross attention module and get the feature representation D ins .

[0015] 4. Large Language Model Text Generation

[0016] For each set of character interaction behavior labels, two sets of text prompts are designed. A large language model (such as GPT3.5) is used to generate text descriptions of the action features of the interactive human body and feature descriptions of the objects in the interactive behavior. The CLIP text encoder is used to encode the text to obtain text features.

[0017] 5. Cross-modal fusion

[0018] The interactive behavior features are cross-modally fused with the text features. The text features containing visual information and the visual features containing text information are updated. This is conducive to training to obtain a well-aligned visual-text space. The updated text features are used to calculate the cosine similarity with the interactive behavior features to obtain the final interactive behavior classification results.

[0019] A second aspect of the present invention relates to a character interaction detection system based on cross-modal fusion of a large language model, comprising:

[0020] Image feature extraction module, used to extract global features of images;

[0021] The multi-task feature extraction module extracts human features, object features, and interaction features to meet the multi-task requirements of human interaction detection, and is used to predict the location of people, the location and category of objects, and the category of interaction behavior.

[0022] Interaction behavior feature enhancement module, used to enhance the interaction feature representation using person-pair instance features and spatial features;

[0023] Large language model text generation module, used to generate corresponding text descriptions based on pre-designed prompts and extract text features;

[0024] A cross-modal fusion module for bidirectionally enhancing feature representation, using text features to enhance visual features and using the enhanced visual features to enhance text features;

[0025] The third aspect of the present invention relates to a character interaction detection device based on cross-modal fusion of a large language model, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the character interaction detection method based on cross-modal fusion of a large language model of the present invention.

[0026] A fourth aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the method for detecting human interaction based on cross-modal fusion of a large language model of the present invention.

[0027] The present invention adopts a query-based approach to extract multiple sets of features from images to adapt to the feature focus of different tasks, and improves the feature representation of interactive behavior through multi-branch feature fusion and interactive behavior feature enhancement, thereby improving the accuracy and reliability of human interaction detection; at the same time, the powerful language expression ability of the large language model is utilized to obtain rich feature description text, and combined with cross-modal fusion training to align the text-visual space well, a good performance effect of human interaction detection is achieved in the case of few samples.

[0028] The advantages of the present invention are as follows: according to the different focuses on features in different tasks of human interaction detection, a query-based method is used to extract multiple groups of features from an image, and multi-branch feature fusion and interactive behavior feature enhancement are used to improve the feature expression of interactive behavior, thereby improving the accuracy and reliability of human interaction detection; at the same time, a large language model and cross-modal fusion are used to obtain richer descriptive text information, effectively train a well-aligned text-visual space, and achieve good performance of human interaction detection with few samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a network structure diagram of character interaction detection based on cross-modal fusion of a large language model in the present invention.

[0030] Figure 2 It is a schematic diagram of a multi-branch feature fusion and interaction feature enhancement module for character interaction detection based on cross-modal fusion of a large language model of the present invention.

[0031] Figure 3 It is a schematic diagram of a cross-modal fusion module for character interaction detection based on cross-modal fusion of a large language model of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is explained and described clearly and completely below in conjunction with the accompanying drawings.

[0033] Example 1

[0034] Reference Figures 1 to 3 This embodiment relates to a method for detecting human interaction based on cross-modal fusion of a large language model, comprising the following steps:

[0035] 1. Image feature extraction

[0036] The original image I to be detected is input into the convolutional neural network to extract the image feature map. The feature map is input into the 1×1 convolution layer to reduce the channel dimension, and the position code is added to the feature map. The feature map is then expanded into a feature sequence and input into the Transformer encoder to obtain the updated image feature sequence V d .

[0037] 2. Multi-task feature extraction

[0038] Three sets of queries Q are defined for the three tasks of human-object detection and interaction behavior classification. h ,Q o ,Q ins At the same time, position-guided encoding is introduced to assign people, objects, and interactive behavior queries at the same location to a group. Then, with the same image visual feature V dThe positional encoding is fed into N Transformer decoder layers, which output the updated query after self-attention and cross-attention calculations. Q h and O o Input to the regression classification head to predict the bounding box coordinates of people and objects and And O o Input category classification head to predict object category Where K o is the number of object categories.

[0039] 3. Enhanced representation of interactive behavior features

[0040] The visual features of human beings h and the visual features of the object O o Splicing to get the visual features of the person instance According to the coordinates of each group of people and object bounding boxes, the spatial information of each group of people is calculated, including the center position distance, aspect ratio, intersection ratio, etc., to obtain the spatial features Then the visual features O of the person instance pair are P and spatial characteristics S P Input feature fusion module MBF, output updated person instance feature Z P Then Z P Interaction behavior characteristics ins Input decoder, calculate Z P =SelfAttn(Z P ), C ins =CrossAttn(Z P ,O ins ), and obtain the enhanced interactive behavior feature C ins . Use CLIP visual encoder to extract image features V s , C ins After the self-attention calculation is performed, the same as V s Input into the cross attention module and get the feature representation D ins .

[0041] 4. Large Language Model Text Generation

[0042] For each interactive behavior label, two sets of prompt inputs are designed into a large language model (such as GPT3.5). The large language model outputs the subject action feature description and object feature description corresponding to each interactive behavior, and uses the CLIP text encoder to encode the generated text to obtain the text feature T.

[0043] 5. Cross-modal fusion

[0044] The interactive behavior feature in step 3 is represented by D ins Perform cross-modal fusion with the text feature T in step 4. Update the text features containing visual information and the visual features containing text information. Then use the updated text features to calculate the cosine similarity with the interactive behavior features to obtain the final interactive behavior classification result.

[0045] The method of the present invention overcomes the problems of the current human interaction detection methods, such as the lack of rich and comprehensive human interaction feature representation and the failure to align the text-visual space well, and improves the accuracy of human interaction detection and the performance effect in the case of few samples.

[0046] Example 2

[0047] This embodiment relates to a character interaction detection system based on large language model cross-modal fusion, which is used to implement the character interaction detection method based on large language model cross-modal fusion of embodiment 1, including:

[0048] Image feature extraction module, used to extract global features of images;

[0049] The multi-task feature extraction module extracts human features, object features, and interaction features to meet the multi-task requirements of human interaction detection, and is used to predict the location of people, the location and category of objects, and the category of interaction behavior.

[0050] An interactive feature enhancement module is used to enhance the interactive feature representation using the visual features and spatial features of the person pair instances;

[0051] Large language model text generation module, used to generate corresponding text descriptions based on pre-designed prompts and extract text features;

[0052] A cross-modal fusion module for bidirectionally enhancing feature representation, using text features to enhance visual features and using the enhanced visual features to enhance text features;

[0053] Example 3

[0054] The present embodiment relates to a character interaction detection device based on cross-modal fusion of a large language model, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the character interaction detection method based on cross-modal fusion of a large language model of embodiment 1.

[0055] Example 4

[0056] The present embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for detecting human interaction based on cross-modal fusion of a large language model of embodiment 1 is implemented.

[0057] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for detecting human interaction based on cross-modal fusion of a large language model, comprising the following steps: Step 1. Image feature extraction: integrating the deep representation capability of convolutional neural networks and the global context understanding capability of Transformer encoders to extract global features of the image. Step 2. Multi-task feature extraction: Based on the query vector, the Transformer decoder is guided to extract human features, object features, and interactive behavior features from the image, and the extracted human features and object features are input into the multi-layer perceptron to predict the location of people and objects and the category of objects respectively. Step 3: Enhance the interactive behavior feature representation. Use a multi-branch fusion module to fuse the character instance features and spatial features, and use the fused features to enhance the interactive behavior feature representation. Step 4. Large language model text generation: define text prompts and generate description text for each pair of person interaction behaviors through the large language model, including the action feature description text of the person in the interaction pair and the feature description text of the object in the interaction pair; Step 5. Cross-modal fusion, use text features to enhance visual features, and use the enhanced visual features to enhance text features, and train a well-aligned text-visual space.

2. The method for detecting human interaction based on large language model cross-modal fusion as claimed in claim 1, characterized in that: Step 1 includes: First, select an original image I to be detected and input it into the convolutional neural network to extract the image feature map; input the feature map into a 1×1 convolution layer to reduce the channel dimension, and add position encoding to the feature map; then expand the feature map into a feature sequence and input it into the Transformer encoder to obtain the updated image feature sequence V d .

3. The method for detecting human interaction based on large language model cross-modal fusion as claimed in claim 1, characterized in that: Step 2 includes: Define query vectors Q for the three tasks of human-object detection and interaction behavior classification h ,Q o and Q ins , guiding the Transformer decoder to extract human features O from the image respectively h , object features O o and interactive behavior characteristics O ins ; Then the extracted human features and object features are input into the multi-layer perceptron to predict the position of the person respectively The location of the object and the category of the object Where K o is the number of object categories.

4. The method for detecting human interaction based on large language model cross-modal fusion according to claim 1, characterized in that: Step 3 includes: The human features O extracted in step 2 h and object features O o Splice to get the character pair instance feature O P ; Construct the spatial feature S of each group of person pairs based on the predicted coordinate frame information P Then, the two sets of features are input into the multi-branch fusion module to fuse the character pair instance features and spatial features to generate a more expressive and discriminative character pair feature Z P ; then Z P Same as the interactive behavior feature O in step 2 ins Input the interactive feature enhancement module, after self-attention calculation and cross-attention calculation, output the interactive behavior feature representation C that is rich in contextual relationship and spatial position information ins ; Finally, C ins and the image features V extracted using the CLIP visual encoder s Input the character interaction decoder together and output the final interaction feature D ins .

5. The method for detecting human interaction based on large language model cross-modal fusion according to claim 1, characterized in that: Step 4 includes: For each pair of character interaction behavior labels, two sets of text prompts are defined, and a large language model (such as GPT3.5) is used to generate description text for each pair of character interaction behaviors, including the action feature description text of the people in the interaction pair and the feature description text of the objects in the interaction pair. Then, the description text of each pair of character interaction behaviors is encoded through the CLIP text encoder to obtain text feature T.

6. The method for detecting human interaction based on large language model cross-modal fusion according to claim 1, characterized in that: Step 5 includes: Combine the text features T encoded in step 4 with the visual features D in step 3 ins Perform cross-modal fusion; use text features to enhance visual features, and use the enhanced visual features to enhance text features; then calculate the cosine similarity between the updated visual features and the updated text features to predict the interaction category of each group of person interaction pairs.

7. A cross-modal character interaction detection system based on a large language model, characterized in that: include: Image feature extraction module, used to extract global features of images; The multi-task feature extraction module extracts human features, object features, and interaction features to meet the multi-task requirements of human interaction detection, and is used to predict the location of people, the location and category of objects, and the category of interaction behavior. Interaction behavior feature enhancement module, used to enhance the interaction feature representation using person-pair instance features and spatial features; Large language model text generation module, used to generate corresponding text descriptions based on pre-designed prompts and extract text features; A cross-modal fusion module is used to enhance feature representation bidirectionally, using text features to enhance visual features and using the enhanced visual features to enhance text features.

8. A person interaction detection device based on cross-modal fusion of a large language model, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the person interaction detection method based on cross-modal fusion of a large language model as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the method for detecting human interaction based on cross-modal fusion of a large language model as described in any one of claims 1 to 6 is implemented.

Images