Method for carrying out human-object interaction detection on image based on visual language model representation

Abstract

The invention provides a method for performing human-object interaction detection on an image based on visual language model representation. The method comprises the following steps: extracting visual features of a to-be-detected image by using a CLIP visual encoder, and extracting text features of the to-be-detected image by using a CLIP text encoder; performing representation learning on the visual features by using a representation learning module to obtain human-object interaction HOI features; and recognizing a bounding box and an interaction type of a person and an object related to interaction in the to-be-detected image according to the text features and the HOI features through an interaction detection module. According to the method, the generalization ability of a new interaction category is enhanced by utilizing priori knowledge of the pre-trained visual language model, knowledge migration from the pre-trained model is realized by exploring representation learning of the pre-trained image and the text, and the ability of the model in processing complex visual contents related to human-object interaction is enhanced. According to the method, knowledge migration from the pre-trained model is realized, and the capability of the model in processing complex visual contents related to human-object interaction is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular, to a method for human-object interaction detection of images based on visual language model representation. Background Art

[0002] HOI (Human Object Interaction) detection is crucial for scene understanding. Its goal is to locate humans and objects in an image and determine the interaction between them - specifically, to identify <human, verb, object> triples. This vision task is crucial in applications such as intelligent monitoring, assisted driving, and healthcare. In recent years, many studies have explored different HOI detection methods. However, most existing methods rely on predefined vocabularies, which makes their adaptation in open-vocabulary scenarios challenging. The HOI categories in the real world are diverse, and it is impossible for a dataset to cover all possible interactions. Open-vocabulary HOI detection attempts to identify categories that were not encountered during the training process.

[0003] Previous HOI detection methods mainly relied on pure vision methods, which lack semantic context information and limit the model's ability to learn new knowledge. The development of large-scale vision-language models now makes more effective open-vocabulary HOI detection possible. In recent years, the rapid progress of computer vision and natural language processing has promoted significant progress in multimodal learning. In particular, the CLIP (Contrastive Language-Image Pre-Training) model, which combines visual and language understanding, has received extensive attention for its ability to learn a shared visual-language representation space. The model efficiently aligns images with text, demonstrating broad generalization capabilities in various tasks such as object detection, semantic segmentation, and image-text generation. Its powerful performance covers multiple fields. Subsequently, more and more studies have begun to apply the dual knowledge of CLIP to human-object interaction (HOI) detection, which is crucial for discerning complex relationships in a scene.

[0004] Currently, the open-vocabulary HOI detection methods in the prior art use simple interaction labels as natural language supervision and utilize the basic knowledge of the CLIP model to detect more HOI categories. Recently, some scholars have further improved the detection accuracy and generalization ability by combining the body part state descriptions generated by large models through fine-grained semantic enhancement.

[0005] The disadvantages of the open-vocabulary HOI detection methods in the prior art include: existing methods mainly rely on independent class labels and ignore the crucial role of HOI text descriptions in context understanding. Existing zero-shot methods initialize the classifier by using HOI text representations, but often fail to fully utilize semantic information, especially in complex scenarios with diverse interactions. In addition, learning HOI knowledge usually requires training with partial sample data. Fine-tuning the CLIP model under such limited samples may lead to catastrophic forgetting - that is, the model loses the knowledge previously acquired when learning new information - and overfitting, that is, the model may be overly adapted to the noise and details in the training data. This over-adaptation may cause the model to misclassify new categories as known categories and fail to detect new categories outside the predefined ones, thus reducing its overall generalization ability.

[0006] In terms of representation learning, most existing methods use pre-trained object detectors to generate regional features of people and objects as the visual features of the image, ignoring the context information of the image. Therefore, fine-grained representations of people and objects are lacking. Summary of the Invention

[0007] Embodiments of the present invention provide a method for detecting human-object interaction in an image based on visual language model representation to effectively detect the interaction relationship between people and objects in the image.

[0008] To achieve the above object, the present invention adopts the following technical solutions.

[0009] A method for detecting human-object interaction in an image based on visual language model representation, comprising:

[0010] Extracting visual features of the image to be detected by using a contrastive language-image CLIP visual encoder, and extracting text features of the image to be detected by using a CLIP text encoder;

[0011] Performing representation learning on the visual features by using a representation learning module to obtain human-object interaction HOI features;

[0012] Identifying the bounding boxes and interaction categories of people and objects related to the interaction in the image to be detected by an interaction detection module according to the text features and the HOI features.

[0013] Preferably, the extracting visual features of the image to be detected by using a CLIP visual encoder includes:

[0014] Construct a feature extraction module including a CLIP visual encoder and a CLIP text encoder, convert the image to be detected into a fixed size, and perform normalization processing. Extract HOI description words and HOI labels from the HOI dataset based on the image to be detected. The HOI labels are used to define specific human-object interaction relationships, and the HOI description words are extended based on the HOI labels.

[0015] Input the processed image to be detected into the feature extraction module. The CLIP visual encoder in the feature extraction module extracts local features through block embedding and encodes the local features into global visual features. The feature extraction module transmits the global visual space features to the representation learning module and the interaction detection module.

[0016] Preferably, extracting the text features of the image to be detected using the CLIP text encoder includes:

[0017] Input the HOI description words and HOI labels to be detected into the CLIP text encoder in the feature extraction module. The CLIP text encoder extracts description features from the HOI description words and label features from the HOI labels. The description features and label features form the HOI text features. The feature extraction module transmits the description features, the label features, and the HOI text features to the interaction detection module.

[0018] Preferably, performing representation learning on the visual features using the representation learning module to obtain human-object interaction HOI features includes:

[0019] Input the global visual space features into the representation learning module. The representation learning module consists of an HOI representation learner and an HOI decoder. The HOI representation learner L(x) consists of two linear transformation layers, and a ReLU activation function is added after each linear transformation layer. The first linear transformation layer compresses the number of channels c1 of the input global visual space feature V I by a ratio r, and the second linear transformation layer then restores the compressed global visual space feature to the original dimension and outputs the enhanced visual feature V' I . Use residual connection to fuse the enhanced visual feature V' I with the global visual space feature V I to obtain a fused feature . The fusion ratio is controlled by the hyperparameter γ. The fused feature passes through a projection layer to obtain the global context visual feature V g . Input the HOI query Q inter and the global context visual feature V g into the HOI decoder to obtain the HOI feature V hoi . The HOI query Q interObtained by the learnable query generation module, the above processing process is expressed by the following formula:

[0020]

[0021] V′ I = L(V I )#(2)

[0022]

[0023] V hoi = HOIDec(V g , Q inter )#(5).

[0024] Preferably, the interactive detection module identifies the bounding boxes and interaction categories of the people and objects related to the interaction in the to-be-detected image according to the text features and the HOI features, including:

[0025] The interactive detection module consists of a bounding box regression branch and an interaction classification branch. Input the HOI feature V hoi into the bounding box regression branch, and this bounding box regression branch uses a multi-layer perceptron MLP for bounding box regression and outputs the bounding box coordinates of the people and objects related to the interaction;

[0026] Input the HOI feature V hoi and the HOI text feature into the interaction classification branch. The HOI text feature includes a description feature and a label feature Calculate the normalized HOI visual feature, calculate the dot product between the normalized description feature and the label feature , and then calculate the interaction classification score S inter ;

[0027] The calculation process of the interaction classification score S inter is as follows:

[0028]

[0029]

[0030] where V′ hoi , and respectively represent the normalized HOI feature, description feature, and label feature matrices, and the variables s1 and s2 are scaling factors;

[0031] The bounding boxes of the people and objects output by the bounding box regression branch and the interaction classification score S interTogether form the final detection result: HOI Detection = {(B human , B object , S inter )}

[0032] B human , B object respectively represent the bounding boxes of the person and the object, and S inter represents the classification score of a certain interaction relationship between the person and the object.

[0033] It can be seen from the technical solutions provided by the embodiments of the present invention described above that the method of the present invention utilizes the prior knowledge of the pre-trained vision-language model to enhance the generalization ability of new interaction categories, and through exploring the representation learning of the pre-trained images and texts, efficiently realizes the knowledge transfer from the pre-trained model, and enhances the model's ability to process complex visual content related to human-object interaction.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description, which will become apparent from the following description, or will be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 is the model framework structure diagram of a method for human-object interaction detection of images based on vision-language model representation provided by an embodiment of the present invention;

[0037] Figure 2 is the processing flow chart of a method for human-object interaction detection of images based on vision-language model representation provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] The following details the embodiments of the present invention. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention, and cannot be construed as a limitation of the present invention.

[0039] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present invention means the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The phrase "and / or" used herein includes any unit and all combinations of one or more of the associated listed items.

[0040] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined as here.

[0041] For the convenience of understanding the embodiments of the present invention, the following will further explain and illustrate with several specific embodiments in conjunction with the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0042] The present invention proposes a method for human-object interaction detection of images based on visual language model representation learning, which is used to improve the generalization ability of the human-object interaction detection model in the open vocabulary scenario. Among them, in the text feature extraction module, HOI text description is introduced as natural language supervision, and the text feature representation is enhanced by jointly using interaction labels to enrich the context semantic information. Aiming at the problems of catastrophic forgetting and overfitting, a HOI feature representation learning module is introduced, which combines a lightweight and trainable linear layer to acquire new knowledge. This representation learner is crucial for effectively transferring pre-trained knowledge, and it can seamlessly integrate the pre-trained visual features of CLIP and the fine-tuned HOI visual features. By minimizing parameter adjustment, the possibility of overfitting is significantly reduced. While retaining the core advantages of the CLIP model pre-trained on a large-scale dataset, this method optimizes it for specific HOI tasks, ensuring strong generalization ability for new tasks.

[0043] The experimental environment to be configured for the present invention includes installing the Linux operating system, Pycharm software, and the PyTorch deep learning framework. In addition, since the training process of the deep learning model takes a long time, it is best to perform the training under the GPU. During the development process, attention needs to be paid to the code implementation of the forward propagation formula corresponding to the loss function. The most difficult to control during the entire development process is the setting of some hyperparameters, such as the thresholds of different modules, the values of the learning rate, the number of iterations, etc. The setting of these parameters has a great impact on the experimental results.

[0044] The model framework structure of a method for human-object interaction detection of images based on visual language model representation provided by an embodiment of the present invention is as Figure 1 shown, including a feature extraction module, a representation learning module, and an interaction detection module. Among them, the feature extraction module includes a CLIP visual encoder and a text encoder, and the input of the text encoder is the HOI descriptor and an independent interaction label. The representation learning module includes a trainable linear layer and a knowledge integration operation. The interaction detection module includes a bounding box regression branch and an interaction classification branch.

[0045] The processing flow of a method for human-object interaction detection of images based on visual language model representation provided by an embodiment of the present invention is as Figure 2 shown, including the following processing steps:

[0046] Step S10: Use the CLIP visual encoder to extract the visual features of the image to be detected, and use the CLIP text encoder to extract the text features of the image to be detected.

[0047] Construct a feature extraction module including a CLIP visual encoder and a CLIP text encoder. The CLIP model of the CLIP visual features and the CLIP text features is initialized using pre-trained parameters, and the parameter initialization process only needs to be performed once.

[0048] Convert the image to be detected into a fixed size and perform normalization processing. Based on the image to be detected, extract and further design the HOI descriptors and HOI labels to be detected from existing HOI datasets (HICO-DET, SWIG-HOI). The HOI label is represented in the form of "verb + object category", such as "riding bicycle", which is used to define the specific human-object interaction relationship. The HOI descriptor is extended on the HOI label through manual design to generate a more linguistically rich natural language description, such as "a person riding a bicycle", to enhance the expression ability of the text features. The HOI descriptor and the HOI label are directly related to the image to be detected and are matched in the common embedding space of the CLIP visual encoder and the CLIP text encoder to ensure the alignment of the visual features and the text description.

[0049] The processed image to be detected is input into the feature extraction module. The CLIP visual encoder in the feature extraction module extracts local features through patch embedding and further encodes them into global visual features. The feature extraction module transmits the global visual spatial features to the representation learning module and the interaction detection module.

[0050] The HOI description words and HOI labels to be detected are input into the CLIP text encoder in the feature extraction module. The CLIP text encoder extracts description features from the HOI description words and label features from the HOI labels. The above description features and label features constitute the HOI text features. The feature extraction module transmits the HOI text features to the interaction detection module. During text encoding, artificially designed HOI description words are additionally introduced to obtain richer context information.

[0051] Step S20: Use the representation learning module to perform representation learning on the visual features to obtain the human-object interaction HOI features.

[0052] This module is responsible for representation learning and knowledge integration from the extracted visual features to achieve effective generalization of new categories. First, the representation learning module realizes fast learning of new knowledge through a lightweight linear layer. Then, by integrating the prior knowledge of the pre-trained CLIP model, the problems of overfitting and catastrophic forgetting are avoided.

[0053] The extracted visual spatial features are first input into the representation learner for knowledge integration. The resulting fused features then pass through a projection layer to obtain global context visual features. To extract visual features related to interactions, the present invention introduces HOI queries. These queries and the projected global context visual features are input into the HOI decoder together. The decoder then generates a visual representation of the interaction and locates the human as the subject of the interaction.

[0054] The representation learning module consists of an HOI representation learner and an HOI decoder. The input data of the representation learning module is the global visual spatial features generated by the CLIP visual encoder, and the output data is the HOI features. The global visual spatial features first undergo feature enhancement through the HOI representation learner L(x). The HOI representation learner L(x) consists of two linear transformation layers, and a ReLU activation function is added after each linear transformation layer to ensure the non-linear expression ability of the features. The first linear transformation layer compresses the number of channels c1 of the input global visual spatial features by a ratio r, while the second linear transformation layer restores the compressed global visual spatial features to the original dimension, and finally outputs the enhanced visual features V'. I Then, the enhanced visual features V' are connected using a residual connection. IFused with the original CLIP visual features to obtain fused features The fusion ratio is controlled by the hyperparameter γ to balance the CLIP prior knowledge and the newly learned features. Subsequently, the fused features Pass through the projection layer to obtain the global context visual feature V g The global context visual feature V g Aims to capture the global information of the entire image to provide a more compact and semantically rich global feature representation. Finally, the HOI query Q inter And the global context visual feature V g Are input into the HOI decoder to obtain the HOI feature V hoi . Among them, the HOI query Q inter Is obtained through a learnable query generation module, which initializes a set of trainable query vectors and adaptively adjusts them during training to better capture the features of the interaction relationship. The above process can be expressed by the following formula:

[0055]

[0056] V′ I = L(V I ) #(2)

[0057]

[0058] V hoi = HOIDec(V g , Q inter ) #(5)

[0059] Step S30: Identify the bounding boxes and interaction categories of the people and objects related to the interaction in the image to be detected according to the text features and the HOI features through the interaction detection module.

[0060] The interaction detection module receives the HOI text features output by the feature extraction module and the HOI feature V output by the representation learning module hoi . The HOI text features include the description features And the label features Calculate the normalized HOI visual features, calculate the dot product between the normalized description features And the label features , and then calculate the interaction classification score s inter .

[0061] The interaction detection module is used to identify and classify interactions between humans and objects. The interaction detection module consists of two branches: a bounding box regression branch and an interaction classification branch. The bounding box regression branch is responsible for locating the bounding boxes of humans and objects related to the interaction in the image. The interaction classification branch classifies the interaction categories based on the visual-text joint features and generates classification results.

[0062] The bounding box regression branch uses an MLP (Multilayer Perceptron) for bounding box regression in order to directly extract the spatial information related to the interaction and perform localization. The input data for this branch is the HOI feature V hoi , and the output data is the bounding box coordinates of the human and the object. The interaction classification branch effectively integrates visual and text information to enhance the accuracy of interaction detection. The input data for this branch is the HOI feature and the HOI text features (description feature, label feature), and the output data is the interaction classification score.

[0063] The interaction visual representation is passed to two different branches. The bounding box regression branch is responsible for regressing the bounding boxes of the human and the object. At the same time, the interaction classification branch projects the interaction visual features into the CLIP pre-trained visual-text space and calculates the cosine similarity between the L2-normalized text features and the interaction visual features. The final interaction classification score is obtained.

[0064] The interaction classification score S inter is calculated as follows:

[0065]

[0066] where V′ hoi , and represent the L2-normalized HOI feature, description feature, and label feature matrices respectively. The variables s1 and s2 are scaling factors used to adjust the range of the calculated similarity scores.

[0067] represent the description feature and label feature obtained by processing through the CLIP text encoder respectively. The input data for the bounding box regression branch is the HOI feature V hoi output by the representation learning module, and the input data for the interaction classification branch is the HOI text feature output by the feature extraction module and the HOI feature V hoi output by the representation learning module.

[0068] When calculating the interaction classification score s inter , the normalized HOI visual feature is calculated respectively with the normalized description feature and the label feature The dot product between them. Since after L2 normalization, the norm of the feature vector is 1, the dot product calculation is actually the cosine similarity, that is, the cosine similarity between the normalized text features and visual features is calculated.

[0069] The calculation method of L2 normalization is as follows:

[0070]

[0071] Among them, ||V||2 represents the L2 norm of the feature matrix (that is, the Euclidean norm of each vector). V′ hoi , and respectively represent the HOI feature, description feature, and label feature matrices after L2 normalization. These normalized features are then used for dot product calculation to obtain the interaction classification score.

[0072] The bounding boxes of the person and object output by the bounding box regression branch and the interaction classification score S output by the interaction classification branch inter together constitute the final detection result: HOIDetection = {(B human ,B object ,S inter )}

[0073] B human ,B object respectively represent the bounding boxes of the person and object, and S inter represents the classification score of a certain interaction relationship between the person and object.

[0074] Because it is necessary to utilize the joint vision-text space of the vision-language pre-training model to capture the relationship between visual information and semantic information. First, the visual encoder and text encoder of CLIP are adopted to generate corresponding visual features and text features, and then the visual features and text features are projected into the vision-text space pre-trained by CLIP to obtain the interaction classification result.

[0075] Because to accurately locate the person and object with interaction, it is necessary to input the interaction representation into the interaction detector and use the interaction visual information and context semantic information for bounding box regression and interaction classification.

[0076] In summary, the embodiment of the present invention designs a method for detecting human-object interaction in images based on vision-language representation learning. This method utilizes the prior knowledge of the pre-trained vision-language model to enhance the generalization ability of new interaction categories, and through exploring the representation learning of pre-trained images and texts, efficiently realizes the knowledge transfer from the pre-trained model, and enhances the model's ability to process complex visual content related to human-object interaction.

[0077] Since many current technologies for learning HOI knowledge usually require using some sample data for training. Fine-tuning a pre-trained vision-language model with such limited samples may lead to catastrophic forgetting and overfitting, that is, the model loses the knowledge previously acquired when learning new information or over-adapts to the noise and details in the training data. Therefore, the present invention designs a representation learning module, which uses a learnable linear layer to learn novel feature representations and integrates the prior knowledge of pre-training, improving the model's ability to understand and classify new interactions.

[0078] Those of ordinary skill in the art can understand that: The drawings are only schematic diagrams of an embodiment, and the modules or processes in the drawings are not necessarily essential for implementing the present invention.

[0079] From the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0080] Each embodiment in this specification is described in a progressive manner. The same or similar parts among the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are only illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0081] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for human-object interaction detection of images based on visual language model representation, characterized in that, Including: Extracting visual features of the image to be detected using a contrastive language-image CLIP visual encoder, and extracting text features of the image to be detected using a CLIP text encoder; Performing representation learning on the visual features using a representation learning module to obtain human-object interaction (HOI) features; Identifying bounding boxes and interaction categories of people and objects related to interaction in the image to be detected according to the text features and the HOI features through an interaction detection module.

2. The method according to claim 1, wherein The extracting visual features of the image to be detected using a CLIP visual encoder includes: Constructing a feature extraction module including a CLIP visual encoder and a CLIP text encoder, converting the image to be detected into a fixed size, and performing normalization processing. Extracting HOI description words and HOI labels from the HOI dataset based on the image to be detected, where the HOI labels are used to define specific human-object interaction relationships, and the HOI description words are extended based on the HOI labels; Inputting the processed image to be detected into the feature extraction module. The CLIP visual encoder in the feature extraction module extracts local features through block embedding, encodes the local features into global visual features, and the feature extraction module transmits the global visual spatial features to the representation learning module and the interaction detection module.

3. The method according to claim 2, wherein The extracting text features of the image to be detected using a CLIP text encoder includes: Inputting the HOI description words and HOI labels to be detected into the CLIP text encoder in the feature extraction module. The CLIP text encoder extracts description features from the HOI description words and label features from the HOI labels. The description features and the label features form HOI text features, and the feature extraction module transmits the description features, the label features, and the HOI text features to the interaction detection module.

4. The method according to claim 3, wherein The performing representation learning on the visual features using a representation learning module to obtain human-object interaction (HOI) features includes: Input the global visual space features into the representation learning module, which consists of an HOI representation learner and an HOI decoder. The HOI representation learner L(x) consists of two linear transformation layers, and a ReLU activation function is added after each linear transformation layer. The first linear transformation layer compresses the number of channels c1 of the input global visual space features V I by a ratio r, and the second linear transformation layer then restores the compressed global visual space features to the original dimension, outputting the enhanced visual features V′ I . The enhanced visual features V′ I are fused with the global visual space features V I using residual connection to obtain the fused features . The fusion ratio is controlled by the hyperparameter γ. The fused features pass through a projection layer to obtain the global context visual features V g . The HOI query Q inter and the global context visual features V g are input into the HOI decoder to obtain the HOI features V hoi . The HOI query Q inter is obtained through a learnable query generation module. The above processing process is expressed by the following formula: V′ I = L(V I )#(2) V hoi = HOIDec(V g , Q inter ) #(5).

5. The method according to claim 4, wherein The identifying bounding boxes and interaction categories of people and objects related to interaction in the image to be detected according to the text features and the HOI features through an interaction detection module includes: The interaction detection module consists of a bounding box regression branch and an interaction classification branch, and inputs the HOI feature V hoi into the bounding box regression branch, which uses a multi-layer perceptron (MLP) for bounding box regression and outputs the bounding box coordinates of the person and object related to the interaction; Input the HOI feature V hoi and the HOI text feature into the interaction classification branch, where the HOI text feature includes a description feature and a label feature Calculate the normalized HOI visual feature, and calculate the normalized description feature and the label feature Take the dot product between them, and then calculate the interaction classification score S inter ; Interaction classification score S inter The calculation process is as follows: where V′ hoi , and represent the normalized HOI feature, description feature, and label feature matrices respectively, and the variables s1 and s2 are scaling factors; The bounding boxes of people and objects output by the bounding box regression branch and the interaction classification score S output by the interaction classification branch inter together constitute the final detection result: HOIDetection = {(B human , B object , S inter )} B human , B object represent the bounding boxes of people and objects respectively, and S inter represents the classification score of a certain interaction relationship between people and objects.

Images