A human-object interaction detection method based on AutoHOINet

Abstract

The application discloses a human-object interaction detection method based on AutoHOINet, comprising the following steps: obtaining the visual modal vector of the to-be-detected image from the to-be-detected image; through a candidate image construction module, extracting the human object and the object object in the to-be-detected image, and reconstructing the same to generate a candidate image; using a human latent relationship mining module, performing human correlation analysis and latent relationship mining to find various latent interaction relationships between the human object and the object object; with the help of a human relationship reasoning module, screening out target relationships from the latent relationships to generate pseudo labels to guide the learning process of the HOI model; finally, using the generated pseudo labels to guide the learning of the HOI model, so as to realize the detection of human interaction. The application can be better than the current weakly supervised HOI model using image-level interaction labels on two benchmark datasets without using artificial labeled HOI label data and weak supervision.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of human-object interaction detection, relates to the fields of artificial intelligence and image processing, and particularly relates to a human-object interaction detection method based on AutoHOINet. BACKGROUND

[0002] In the field of computer vision, human-object interaction detection (HOI) refers to the identification of the exact positioning of a human and an object in an image and the corresponding interaction category by a computer to achieve in-depth understanding of human activities. Specifically, the HOI detection task accepts an image and outputs a series of triplets (<human, object, interaction>). Therefore, the success or failure of the task is affected by multiple key factors, including the accurate positioning of the human object and the object, the correct classification of the object category, and the accurate grasp of the interaction relationship between the human and the object.

[0003] The supervised learning based human-object interaction detection task relies on the detailed labels of <person, object, action> annotated by human, which are used for the training of the network. Currently, the supervised learning HOI detection task is divided into two main branches: two-stage and one-stage methods. Two-stage methods (such as PPDM proposed by Liao et al., DRG proposed by Gao et al.) first use a pre-trained object detection network to detect people and objects in the image, and then pair each person and object and send it into the interaction discrimination network to complete the interaction detection. Two-stage models are relatively simple and easy to understand, but in the interaction discrimination stage, they will pair the people and objects detected in the first stage one by one, so the running speed is slow and the efficiency is low. Therefore, one-stage HOI detection based on the Transformer architecture has gradually developed. HOITrans proposed by Zou et al. and QPIC proposed by Tamura et al. use the extractor and encoder in the DETR architecture to extract features and obtain global feature encoding, truly realizing end-to-end HOI detection. Recently, more and more architectures that integrate CLIP as additional text information have emerged, such as GEN-VLKT proposed by Liao et al. and PBRL (WEAKLY-SUPERVISED HOI DETECTION VIA PRIOR-GUIDED BI-LEVEL REPRESENTATION LEARNING) proposed by Wan et al.

[0004] Weakly-supervised HOI detection (Weakly-HOI) usually uses image-level interaction labels for training. Specifically, given an image and the interaction relationship that exists in it (but not including the specific interaction relationship between people and objects), the network determines which people and objects actually have an interaction relationship through autonomous learning. Compared with supervised learning, weakly-supervised methods only need image-level labeling, without specific labeling of person position and interaction action, thus greatly reducing the cost of manpower.

[0005] Currently, the human-object interaction detection method based on weak supervision can be divided into two categories: one uses <interaction, object> labels, and the other only uses <interaction>Labels. MX-HOI model proposed by Kumaraswamy et al. proposes a momentum independent learning architecture on the basis of simultaneously using weakly supervised labels and strongly supervised labels. On the other hand, AlignFormer proposed by Kilickaya et al. realizes the pseudo-alignment training strategy by introducing the align layer. However, these methods all face the problem of extremely noisy person-object association and ambiguous interaction types.

[0006] At present, whether the person interaction detection method based on full supervision or weak supervision learning needs a large amount of artificial labeled data, thereby consuming a large amount of time and manpower. Although the existing technology has tried to analyze the person relationship based on weakly labeled data, the problem is still challenging. SUMMARY

[0007] The purpose of the present application is to propose a person-object interaction detection method based on AutoHOINet, which can complete HOI detection without artificial labeled labels, thereby solving the problem of high labeling cost of HOI dataset. AutoHOINet is composed of four components: HOI model, candidate image construction module, person latent relationship mining module and person relationship reasoning module. Specifically, the HOI model obtains its visual modal vector from the image to be detected. The candidate image construction module detects the possible person and object in the image through the target detection framework, and denoises and recombines in space. The person latent relationship mining module uses the adaptability of the large-scale language model CLIP (Contrastive Language-Image Pre-training) in the target domain to analyze the person association and mine the latent relationship, and discover various latent interaction relationships between the person object and the object. The person relationship reasoning module combines the image-text similarity vector with the prior knowledge mask to remove other interaction behavior interference information and generate pseudo labels. Finally, the generated pseudo labels are used to guide the learning of the HOI model, thereby realizing the detection of person interaction.

[0008] The present application is realized at least by one of the following technical solutions.

[0009] A person-object interaction detection method based on AutoHOINet, comprising the following steps:

[0010] a. extracting the visual modal vector of the image to be detected by the HOI model;

[0011] b. inputting the image to be detected into the candidate image construction module, extracting the visual features by the convolutional neural network, obtaining the specific position of the person object and the object in the image to be detected, and reconstructing the same to generate a candidate image;

[0012] c. The candidate image is subjected to person correlation analysis and potential relationship mining by the person potential relationship mining module, and various potential interaction relationships between the person object and the object object are explored;

[0013] d. The target relationship is filtered out from the potential relationship by means of the person relationship reasoning module, and pseudo-labels are generated to guide the learning of the HOI model;

[0014] e. The generated pseudo-labels and related loss functions are used to guide the learning of the HOI model, thereby realizing the detection of person interaction.

[0015] Further, the HOI model is composed of GEN-VLKT, and the image to be detected wherein, represents the matrix dimension, H, W and C represent the height, width and channel number of the image to be detected, respectively.

[0016] Further, the backbone network (Backbone) of the GEN-VLKT is a pre-trained ResNet-50.

[0017] Further, the GEN-VLKT is a model combining a guided-embedding network (GEN) and a visual-linguistic knowledge transfer (VLKT) training strategy, wherein the GEN is used to solve the person-object association problem, and the VLKT is used to enhance the interaction understanding ability.

[0018] Further, the ResNet-50 includes a 7×7 convolution kernel, a 3×3 pooling layer, and a backbone network for feature extraction composed of four residual groups, wherein the first residual group has a total of 3 residual blocks, and each residual block has 3 layers of 1×1×64, 3×3×64, and 1×1×256 convolution kernels, respectively; the second residual group has a total of 4 residual blocks, and each residual block has 3 layers of 1×1×128, 3×3×128, and 1×1×512 convolution kernels, respectively; the third residual group has a total of 6 residual blocks, and each residual block has 3 layers of 1×1×256, 3×3×256, and 1×1×1024 convolution kernels, respectively; and the fourth residual group has a total of 3 residual blocks, and each residual block has 3 layers of 1×1×512, 3×3×512, and 1×1×2048 convolution kernels, respectively.

[0019] Further, the convolutional neural network in step b is Yolov8, which is used to locate the person object and the object object, and then uses an instance segmentation strategy to generate candidate images.

[0020] Further, the output result p = (c x ,c y ,W x ,H y ,c) of Yolov8, wherein c x ,c y represents the center point coordinates of the bounding box (Bounding boxes), W x ,H y represents the width and height of the bounding box, and c represents the category of the instance.

[0021] Further, the instance segmentation strategy extracts the region of interest from the image to be detected according to the Bounding boxes and the center point coordinates. With this strategy, the influence of other possible noise can be reduced.

[0022] Further, the person potential relationship mining module includes a pre-trained model based on contrastive text-image pairs (Contrastive Language–Image Pre-training, CLIP).

[0023] Further, the CLIP mainly includes two modules of text encoding (Text Encoder) and image encoding (Image Encoder), which are responsible for extracting text and image features respectively. In addition, by calculating the similarity of the two modalities, the model can gradually learn the matching relationship between text and image.

[0024] Further, the similarity calculation involves the cosine similarity between image features and text features. The cosine similarity has a value range of -1 to 1. The closer the value is to 1, the more similar the two vectors are. The closer the value is to -1, the less similar the two vectors are. A value of 0 indicates that the two vectors are completely unrelated.

[0025] Further, the person relationship reasoning module in step d is composed of action correlation matching, mask screening based on prior knowledge, dynamic threshold selection, and interactive action screening.

[0026] Further, the action correlation matching scheme adopts a Top-One strategy, which selects the interactive behavior with the highest text-image similarity as the initial interactive action.

[0027] Further, the mask screening scheme based on prior knowledge is used to screen suitable candidate verbs for object objects.

[0028] Further, the candidate verb refers to all possible actions performed by the object in the candidate image.

[0029] Further, the dynamic threshold selection scheme is used to evaluate whether the human-object interaction occurs in the image. The method can flexibly adjust the threshold according to the actual situation, improve the effective detection and recognition ability of the interaction relationship, adapt to different scenes, and more accurately determine whether the interaction event occurs.

[0030] Further, the interaction action screening scheme is used to screen the target action from the N T candidate interaction actions.

[0031] Further, in step e, the loss function is

[0032]

[0033] wherein respectively represent the classification loss, the bounding box regression loss, and the IoU loss of the human-object bounding box, and ρ, q, and r respectively represent the classification, regression, and intersection-over-union candidate scenarios, and h, o, and a respectively represent the human, object, and action, and α ρ , α q , α r , and β1-β4 are hyperparameters, represent the loss of the CLIP model.

[0034] Compared with the prior art, the present application has the following beneficial effects:

[0035] 1. The human-object interaction detection method based on AutoHOINet proposed in the present application is the first method to complete HOI detection without manual annotation.

[0036] 2. The present application fully utilizes the adaptive ability of the model in the target field, effectively reduces the cost of human and material resources, thereby significantly reducing the burden of HOI data annotation.

[0037] 3. Through experimental comparison, the method proposed in the present application exhibits excellent performance, not only surpassing the current methods using image-level interaction labels (weakly+, weakly, weakly-), but also achieving significant advantages in comparison with the performance of early full-supervised models. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is the overall network framework diagram of the human-object interaction detection method based on AutoHOINet of the embodiments of the present application;

[0039] Figure 2 A schematic diagram of a to-be-detected image of an embodiment of the present application is shown in the figure.

[0040] Figure 3 A framework diagram of a person potential relationship mining module of an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0041] In order to more clearly set forth the objects, technical solutions and advantages of the present application, the present application will be further described in detail below with reference to the accompanying drawings. It should be noted that the present application can be implemented in various ways different from the above description. Therefore, other various embodiments derived by those skilled in the art without creative labor should all fall within the scope of protection covered by the present application.

[0042] As shown in the figure, the present embodiment provides a human-object interaction detection method based on AutoHOINet, which comprises the following steps: Figure 1

[0043] Step 1: extracting a visual modal vector from a to-be-detected image by using a HOI model;

[0044] The to-be-detected image refers to an image that needs to be detected. In order to explain the human interaction detection method provided by the present embodiment in detail, a three-by-three image is used for illustration.

[0045] As a preferred embodiment, as shown in the figure, the to-be-detected image includes three men and three bicycles. Figure 2

[0046] Specifically, the to-be-detected image is input into the AutoHOINet, wherein, represents the matrix dimension, and H, W and C represent the height, width and channel number of the to-be-detected image, respectively.

[0047] As a preferred embodiment, the HOI model in step 1 is composed of GEN-VLKT, which is a model combining a guided-embedding network (GEN) and a visual-linguistic knowledge transfer (VLKT) training strategy. The GEN is used to solve the person-object association problem, and the VLKT is used to enhance the interaction understanding ability.

[0048] Specifically, the backbone network of the GEN-VLKT is a pre-trained ResNet-50. The GEN is composed of a CNN-transformer visual encoder with a transformer encoder structure and two decoders.​​

[0049] The example decoder and the relation decoder constitute the decoder described in GEN in parallel. Each example query vector Q ins The human query vector Q h And the object query vector Q o Is combined, and a position guide vector P q The same position of human and object vectors are associated into a matching pair. Q ins It can be represented by formula 2:

[0050] Q ins = [Q h + P q , Q o + P q ] (2)

[0051] The relation decoder uses the decoding features of the example query vector to guide the relation query, which can be seen in formula 3, where represents the input query vector of the kth relation decoder, represents the example decoding feature of the kth person object, represents the example decoding feature of the kth object object.

[0052]

[0053] Further, the VLKT first uses CLIP text embedding (CLIP Text Embedding) to extract sentence features to initialize the weight parameters of the interaction classifier (Interaction Classifier). Then the adjusted and cropped image is input into the pre-trained CLIP encoder (CLIP Image Encoder) to extract visual features and guide interactive visual feature learning.

[0054] Visual features And CLIP text features Calculate the similarity as the output of the relation classifier

[0055]

[0056] Where θ represents the magnification coefficient, represents the text feature vector, N a represents the number of interaction action categories, represents the action score of the ith image. Finally, the GEN-VLK outputs the human interaction relationship Where h box , o box bounding boxes representing the person object and the object object, respectively, o representing the object category, a h,o representing the action category, representing the action score.

[0057] As a preferred embodiment, the ResNet-50 specific parameter settings are shown in Table 1, including a 7x7 convolution kernel, a 3x3 convolution kernel, and a backbone network for feature extraction composed of 4 residual groups. The first residual group has a total of 3 residual blocks, each with 3 layers of 1x1x64, 3x3x64, and 1x1x256 convolution kernels, a total of 9 layers; the second residual group has a total of 4 residual blocks, each with 3 layers of 1x1x128, 3x3x128, and 1x1x512 convolution kernels, a total of 12 layers; the third residual group has a total of 6 residual blocks, each with 3 layers of 1x1x256, 3x3x256, and 1x1x1024 convolution kernels, a total of 18 layers; the fourth residual group has a total of 3 residual blocks, each with 3 layers of 1x1x512, 3x3x512, and 1x1x2048 convolution kernels, a total of 9 layers.

[0058] Table 1 ResNet-50 parameter settings

[0059]

[0060] Step 2, input the image to be detected into the candidate image construction module, extract visual features through the convolutional neural network, obtain the specific position of the person object and the object object in the image to be detected, and then use the instance segmentation strategy to generate candidate images;

[0061] The convolutional neural network is composed of Yolov8, which is used for positioning the person object and the object object.

[0062] Follow the detector based on Yolov8. Given an input image where H, W, and C represent the height, width, and channel of the input image, respectively, after Yolov8 extraction, the output result p=(c x ,c y ,W x ,H y ,c), where c x ,c y represent the center point coordinates of the bounding boxes (Bounding boxes), W x ,H y represent the width and height of the Bounding boxes, and c represents the category of the instance.

[0063] Person interaction detection is a relatively complex process, not only to detect what the object in the image is, but also to detect what behavior actions the person in the image has done to the object. And the image in reality is often not a one-to-one single relationship, the image often contains multiple person objects and multiple object objects, which will greatly increase the difficulty of detection. Therefore, the instance segmentation strategy described in step 2 extracts the region of interest (i.e., first locates the position of the person object and the object object) from the image to be detected according to the output result p = (c x ,c y ,W x ,H y ,c) of Yolov8.

[0064] Specifically, N h persons and N o objects are paired two by two to generate N I = N h *N o candidate images. With this strategy, the influence of other possible noise can be reduced.

[0065] Step 3, through the person potential relationship mining module, the person correlation analysis and potential relationship mining of the result of step 2 are carried out, and multiple potential interaction relationships between the person object and the object object are explored.

[0066] The person potential relationship mining module is composed of a pre-training model based on contrastive text-image pairs (Contrastive Language-Image Pre-training, CLIP).

[0067] Specifically, as Figure 3 shown, the CLIP mainly includes two modules of text encoding (Text Encoder) and image encoding (Image Encoder), which are respectively responsible for extracting text and image features. In addition, by calculating the similarity of the two modalities, the model can gradually learn the matching relationship between the text and the image.

[0068] CLIP Image Encoder. The N I candidate images constructed are input into the image encoding module of CLIP, and the image encoding of CLIP is generated by using the Image Encoder of CLIP. The image encoding process is represented by formula 5. In the formula, the image encoding feature where C IE is the output dimension of the Image Encoder, so that each row of I E represents the encoding representation of a candidate image, and I candidate represents the candidate image.

[0069] I E = ImageEncoder(I candidate ) (5)

[0070] CLIP Text Encoder. The labels in the dataset are converted into textual descriptions, the main steps are as follows: first, construct the text template T1 of HOI interactive action in the form of "a photo of a person verb-ing an object", such as the triple <human, ride, motorcycle> is converted into "a photo of a person riding a motorcycle". The constructed text template is sent into the Text Encoder for encoding to obtain the text information matrix where N T represents the number of text contained in the text template, i.e. the number of HOI interactive action categories, C TE represents the dimension of text encoding.

[0071] In addition, another kind of text template T2 ("a photo of a person verb-ing") is constructed, which aims to enhance the importance of the verb in the HOI person interactive relationship, in order to further emphasize the attention to the action of the person. T1 and T2 represent the language templates of different HOI actions, and their corresponding text information matrices are T E1 and T E2 , respectively.

[0072] T E1 = TextEncoder(T1) (6)

[0073] T E2 = TextEncoder(T2) (7)

[0074] The two modal similarity calculation involves the cosine similarity between image features and text features.

[0075] Specifically, the image encoding information I E and the text information matrix T E The cosine similarity calculation process is shown in equations (8)-(10):

[0076]

[0077] where, represents the similarity of image encoding information I E and text information matrix T E1 represents the image encoding information I E ​and the similarity of the text information matrix T E2 S represents the comprehensive similarity of each action.

[0078] wherein, respectively represent the image encoding vector and the text encoding vector, and ·T E1 T , ·T E2 T represents the transpose matrix of the vector T E1 , T E2 , and the symbol |||| represents the length of the vector. At this time, the cosine similarity is in the range of -1 to 1, and the value is closer to 1, indicating that the two vectors are more similar, and the value is closer to -1, indicating that the two vectors are less similar, and the value is 0, indicating that the two vectors are completely irrelevant.

[0079] Specifically, in the formula Therefore, by formulas (8)-(10), N I candidate images occur N T types of interactive actions, that is, the image-text similarity of the human-object interaction relationship in each candidate image and the text template.

[0080] Step 4, with the help of a person relationship reasoning module, further encoding the multiple potential interactive relationships between the person object and the object object, filtering the target relationship from the potential relationship, generating a pseudo label to guide the learning of the HOI model;

[0081] In step 4, the person relationship reasoning module is composed of action correlation matching, mask filtering based on prior knowledge, dynamic threshold selection, and interactive action filtering.

[0082] The action correlation matching scheme adopts a Top-One strategy.

[0083] Specifically, for each candidate image in the result S output by the person potential relationship mining module, the present application adopts a Top-One strategy, that is, selects the interactive behavior with the highest image-text similarity as the initial interactive action.

[0084] Considering the characteristics of the HICO dataset, it is found that the occurrence of a certain HOI behavior action is often accompanied by a high probability of occurrence of other similar interactive actions, for example, in the specific scene of "person riding a bike", for the action "riding", other interactive actions including "person sitting on the bike" and "person straddling the bike" are often accompanied by behaviors. Based on this observation, the action correlation matching is used to infer that there is a strong correlation between the Top-One selected interactive action and other behaviors, and the image-text similarity of the other behaviors is amplified.

[0085] Specifically, the present application autonomously promotes their similarity to highlight the relevance and similarity between these interactive actions. Formulas (11) and (12) describe in detail the detailed steps of applying action correlation matching in a candidate image.

[0086] index a =[j1,j2,...,j n ]=ActMatching(S i_max ) (11)

[0087] S ij =S ij *scale,j∈index a (12)

[0088] where S i_max represents the interactive action with the highest image-text similarity in the i-th candidate image, the ActMatching function represents selecting several other behaviors that have strong correlation with S i_max , j n represents the index of the original behavior list selected from the relevant behaviors. Where index a represents the index set of several interactive actions associated with S i_max , scale represents the magnification factor, S ij represents the value of the i-th row and j-th column in the S matrix, and j represents the column index.

[0089] The mask screening based on prior knowledge screens appropriate candidate verbs for the object.

[0090] The candidate verb refers to all possible actions of the object in the candidate image. Certain objects often show obvious relevance to certain action categories.

[0091] As a preferred embodiment, for the object "horse", the commonly relevant interactive actions include "people riding horses", "people leading horses", "people feeding horses", etc., at this time "riding", "leading", "feeding", etc. are the candidate verbs corresponding to the object "horse", and impossible actions include "people throwing horses" or "people eating horses", etc. For example, for the object "apple", the candidate verbs include "holding", "eating", etc., and impossible actions include "riding", "leading", etc. Based on such observations, the mask screening scheme based on prior knowledge adopts a certain mask mechanism to screen the interactive actions of certain objects according to prior knowledge.

[0092] Further, formulas (13)-(15) are used to update the similarity score S ij :

[0093] index o = [j1, j2,..., jN] (17) n ] = PirorFilter(o i ) (13)

[0094]

[0095] S ij = S ij · mask j , j = 1, 2,..., N T (15)

[0096] where o i denotes the object class in the ith image; the PirorFilter function denotes the index of all the interactive actions related to o i ; the index o denotes the index set of the interactive actions related to o i ; m j ∈ {0, 1}, 1 indicates that the ith action is related to o i , and 0 indicates that it is not related; mask denotes the mask of the interactive actions, denotes the element in the list, and mask j denotes the jth element in the mask list.

[0097] Therefore, by adopting this mask mechanism, the behavior actions related to a specific object class are retained, thereby avoiding the interference of other behaviors that do not occur on the object class as much as possible.

[0098] The dynamic threshold selection scheme is used to evaluate whether the human-object in the image has an interactive behavior.

[0099] Further, the calculation formula of the dynamic threshold is as follows:

[0100]

[0101] where max(S i ) denotes the maximum value of the vector S i , ω1 is a parameter for controlling the threshold range, and Thre > 0 indicates that the human and the object in the image have an interactive behavior.

[0102] This method can flexibly adjust the threshold according to the actual situation, improve the effective detection and recognition ability of the interactive relationship to adapt to different scenes, and thus more accurately determine whether the interactive event occurs.

[0103] The interactive action screening scheme is used to screen the target action from the N T candidate interactive actions. Specifically, it can be represented by formula (17):

[0104] a index = {(j) | max(S i ) - max(S i ) x ω2 < S ij < max(S i ), S ij ∈ S i} (17)

[0105] where j represents the serial number of each interactive action, ω2 represents the weight, a index represents the set of action serial numbers. In this way, the position information and the object category information of the previously detected person object and object object constitute the HOI label set y of the person-object pair:

[0106] y = {(h box , o box , c o , a i ) | a i ∈ a index} (18)

[0107] where h box , o box represent the detected person object and the bounding box of the person object, c o represents the object category, and a i represents the interactive action serial number. Thus, the present application generates the HOI label suitable for training without manual annotation.

[0108] Step 5, using the label generated in step 4 and the related loss function to guide the learning of the HOI model, so as to realize the detection of person interaction.

[0109] Further, in order to obtain the optimal network model, the Hungarian algorithm is used to assign a unique prediction to each real label, so that the network input corresponds to the real label, and the calculation formula is shown in (20)-(21):

[0110]

[0111] In formula (19), y represents the HOI label set, N q represents the number of query vectors in the Transformer network, represents the matching loss of y i , , and is the interactive relationship pair predicted by the network, y i represents the i-th real label, represents the i-th prediction of the network. ρ, q, r in formula (20) represent the classification, regression, and intersection-over-union candidate scenes, respectively, h, o, a represent the person, object, and action, respectively, represents the classification loss, represents the regression loss of the position box, represents the IoU loss of the person bounding box; in addition, β1, β2, β3, α p , α q , and α r are hyperparameters of the matching loss (which can be set by the user) and are used to balance each loss. After matching, the overall loss is calculated using formula (21):

[0112]

[0113] wherein represents the loss of the CLIP model, and β4 is a hyperparameter of the matching loss (which can be set by the user).

[0114] The effectiveness of the present application is verified based on benchmark data HICO-Det and V-COCO. The HICO-Det dataset contains 47,776 images, of which 38,118 are used for training and 9,658 are used for testing. The dataset covers 80 object classes and 117 action classes, a total of 600 different interaction types, of which 462 are non-rare (no-rare) classes and 138 are rare (rare) classes. The V-COCO dataset contains 10,326 images, of which 5,400 are used for training and 4,964 are used for testing, a total of 80 object classes and 29 action classes, including 4 body actions that do not involve any object interaction. The HICO-Det and V-COCO datasets cover a wide range of object and action classes, which are of great significance for evaluating the performance of HOI detection algorithms.

[0115] The mean average precision (mAP) is used as an evaluation indicator. The result of the HOI triplet prediction is correct when the following conditions are met: 1) the IoU of the predicted person and object bounding boxes is >0.5; 2) the predicted class is correct. For the HICO-Det dataset, three different class sets are evaluated: all 600 HOI classes (Full), 138 HOI classes with less than 10 training instances (Rare), and the other 462 HOI classes (Non-Rare). For the V-COCO dataset, the role mAP is reported in two scenarios: S1 includes 29 action classes, including 4 body actions; S2 includes 25 action classes, excluding object-free HOI classes.

[0116] Table 2 Comparison results of the present application with other algorithms on the HICO-Det dataset

[0117]

[0118] Table 3 Comparative results of the present application using different HOI models on Weakly+

[0119]

[0120] Table 4 Comparative results of the present application on V-COCO dataset with other algorithms

[0121]

[0122] Tables 2-4 show the comparative results of the present method and other methods. It can be clearly observed from the tables that, relative to other methods, the present method (AutoHOINet) adopts a HOI task framework without manual annotation of labels and achieves good results, which shows that the present method is superior to other methods in performance.

[0123] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The preferred embodiments do not describe all the details and limit the present application to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can well understand and use the present application.< / interaction>

Claims

1. A human-object interaction detection method based on AutoHOINet, characterized in that, The method comprises the following steps: a. extracting a visual modality vector from a to-be-detected image by using an HOI model; b. inputting the visual modality vector extracted from the to-be-detected image into a candidate image construction module, performing visual feature extraction by using a convolutional neural network, obtaining the specific positions of the human object and the object object in the to-be-detected image, and reconstructing to generate a candidate image; c. performing human correlation analysis and potential relationship mining on the candidate image by using a human potential relationship mining module, and exploring multiple potential interaction relationships between the human object and the object object; d. filtering out a target relationship from the potential relationship by using a human relationship reasoning module to generate a pseudo label to guide the learning of the HOI model; in step d, the human relationship reasoning module comprises four parts of action correlation matching, mask screening based on prior knowledge, dynamic threshold selection and interactive action screening; The optimal selection strategy is adopted in the action correlation matching scheme; The mask screening scheme based on prior knowledge is used for screening candidate verbs for the object object; The candidate verb refers to all possible actions performed by the object object in the candidate image; The dynamic threshold selection scheme is used to evaluate whether the human-object in the candidate image has an interaction behavior; The interaction action screening scheme is used to screen the target action from the candidate N T interaction actions. e. guiding the learning of the HOI model by using the generated pseudo label and the related loss function, so as to realize the detection of human interaction.

2. The AutoHOI Net-based human-object interaction detection method according to claim 1, wherein the HOI model comprises a GEN-VLKT, and the image to be detected is an image to be detected. wherein, denotes the matrix dimension, H, W and C denote the height, width and channel number of the image to be detected, respectively.

3. The AutoHOI Net-based human-object interaction detection method according to claim 2, characterized in that, The backbone network Backbone of the GEN-VLKT is a pre-trained ResNet-50.

4. The AutoHOI Net-based human-object interaction detection method of claim 2, wherein GE The N-VLKT is a model combining a guided-embedding network (GEN) and a visual-linguistic knowledge transfer (VLKT) training strategy, wherein the GEN is used to solve the human-object correlation problem, and the VLKT is used to enhance the interaction understanding capability.

5. The AutoHOI Net-based human-object interaction detection method of claim 3, characterized in that, The ResNet-50 includes a backbone network for feature extraction with a 7*7 convolution kernel, a 3*3 pooling layer and four residual groups, wherein the first residual group has three residual blocks, each of which has three layers of 1*1*64, 3*3*64 and 1*1*256 convolution kernels; the second residual group has four residual blocks, each of which has three layers of 1*1*128, 3*3*128 and 1*1*512 convolution kernels; the third residual group has six residual blocks, each of which has three layers of 1*1*256, 3*3*256 and 1*1*1024 convolution kernels; and the fourth residual group has three residual blocks, each of which has three layers of 1*1*512, 3*3*512 and 1*1*2048 convolution kernels.

6. The AutoHOI Net-based human-object interaction detection method of claim 1, wherein, The convolutional neural network is Yolov8, which is used for positioning the human object and the object object and generates a candidate image by using an instance segmentation strategy; The output result p of the Yolov8 is (c x ,c y ,W c ,H y ,c), wherein c x ,c y represents the center point coordinates of the bounding box, W x ,H y represents the width and height of the bounding box, and c represents the category of the instance. The instance segmentation strategy extracts a region of interest from the to-be-detected image according to Bounding boxes and center point coordinates.

7. The AutoHOI Net-based human-object interaction detection method of claim 1, wherein, In step c, the person potential relationship mining module comprises a pre-training model based on contrastive text-image pairs (Contrastive Language-Image Pre-training, CLIP).

8. The AutoHOI Net-based human-object interaction detection method according to claim 7, characterized in that, The CLIP includes a text encoding (Text Encoder) module and an image encoding (Image Encoder) module, which are respectively responsible for extracting text and image features. In addition, by calculating the similarity of the two modalities, the matching relationship between the text and the image can be gradually learned.

9. The AutoHOI Net-based human-object interaction detection method of claim 8, characterized in that, The similarity calculation involves the cosine similarity between the image features and the text features. The cosine similarity has a value range of -1 to 1. The closer the value is to 1, the more similar the two vectors are. The closer the value is to -1, the less similar the two vectors are. A value of 0 indicates that the two vectors are completely unrelated.

Images