Character interaction detection method and device based on cascade parallel, equipment and medium

Abstract

The invention discloses a cascade parallel-based character interaction detection method and device, equipment and a medium, and relates to the technical field of image recognition. The method comprises the following steps: acquiring a to-be-identified image, extracting image features, and simultaneously obtaining position information of the image features; and iterating the following step # imgabs0 # times, and predicting a character interaction detection result according to the output of the last iteration. Generating a human body query vector and an object query vector according to the image features and the figure pair prior bounding box; and decoding according to the human body query vector to obtain a human body decoding feature. And updating an object query vector according to the human body decoding feature, and decoding according to the updated object query vector to obtain an object decoding feature. And obtaining an initial interactive decoding feature according to the human body decoding feature and the object decoding feature. And according to the human body decoding features, the object decoding features and the initial interactive decoding features, obtaining multi-relation features, and then performing fusion. And enhancing the initial interactive decoding feature according to the fused relation feature, and obtaining an interactive feature vector.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method, device, equipment and medium for detecting human interaction based on cascade parallelism. Background Art

[0002] In the field of artificial intelligence, human interaction detection technology has a wide range of application needs in video analysis, intelligent monitoring, human-computer interaction, behavior recognition and other fields. The key technology in these application scenarios lies in whether human interaction behavior can be accurately and efficiently recognized. Accurate human interaction detection can enable computer vision systems to better understand scene semantics, thereby improving the intelligence and practicality of the system.

[0003] To solve the problem of detecting the interactive relationship between human and objects, the existing technologies are mainly divided into two-stage detection methods and one-stage detection methods. The two-stage detection method adopts the strategy of detection first and then classification. It screens out a large number of negative samples through instance detection, and then classifies the interaction behavior of the retained positive samples. Although this method can effectively use additional contextual information to improve the detection results, its serial structure leads to low efficiency, a large amount of computing resources are wasted in the generated proposals, and the inference time is also long. The one-stage detection method regards the human interaction detection task as a whole, performs instance detection and interaction classification in real time through parallel detectors, and merges the prediction results using special post-processing methods. However, this method faces challenges in multi-task learning. Due to the significant differences in the visual features required for human-object detection and interaction classification tasks, the performance of the model is easily limited. In addition, the traditional Transformer-based object detection model regards the object query as a set of learnable embeddings, but these embeddings usually lack clear physical meanings and cannot intuitively explain where the detector should focus, which increases the difficulty in the model optimization process.

[0004] In summary, the existing technologies in the task of human interaction detection have problems such as low efficiency, multi-task learning conflicts and difficulty in model optimization. These problems limit the performance of human interaction detection technology in practical applications. Summary of the invention

[0005] The present invention provides a cascade-parallel-based character interaction detection method, device, equipment and medium to improve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for detecting human interaction based on cascade parallelism, comprising: S01. Obtain an image to be recognized.

[0007] S02. Extracting image features according to the image to be identified, and generating position information of the image features.

[0008] S03, Iteration The following steps are performed, and the human interaction detection results are predicted based on the human feature vector, object feature vector and interaction feature vector output by the last iteration.

[0009] S04, generating a human query vector and an object query vector according to the image features and the prior bounding box of the person pair, and obtaining a query vector group, wherein the prior bounding box of the person pair is obtained by random initialization in the first iteration, and is updated in subsequent iterations according to the human decoding features and object decoding features of the previous iteration.

[0010] S05. According to the image features, the position information and the human query vector, obtain human decoding features through decoding by a human detector.

[0011] S06. Update the object query vector according to the human body decoding feature, and then obtain the object decoding feature by decoding through the object detector according to the updated object query vector, the image feature and the position information.

[0012] S07: Input the image features, the human body decoding features and the object decoding features into a classification decoder to obtain initial interactive decoding features.

[0013] S08. Input the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain unary relationship features, paired relationship features and ternary relationship features.

[0014] S09. Embed the unary relationship feature, the paired relationship feature and the image feature into the ternary relationship feature through an attention module for fusion.

[0015] S10. Enhance the initial interactive decoding features through the fused ternary relationship features to obtain an interactive feature vector.

[0016] In an optional implementation, the person pair prior bounding box includes a human body prior bounding box and an object prior bounding box. The query generator, human body detector and object detector are all stacked, and the number of stacked layers is layer.

[0017] In an optional implementation, step S04 specifically includes steps S041 to S046.

[0018] S041. According to the human body prior bounding box and the object prior bounding box, obtain human body region features and object region features from the image features, and merge the two region features according to feature dimensions to obtain person pair region features.

[0019] S042, according to the regional features of the characters, through the convolution layer plus the multi-layer perceptron The activation functions predict the key points of the human body and the key points of the object respectively.

[0020] S043. According to the key points of the human body and the key points of the object, a bilinear interpolation sampling operation is performed on the regional features of the human body to extract local visual features of the human body and local visual features of the object respectively.

[0021] S044. Aggregate the local visual features of the human body and the local visual features of the object through a fully connected layer to obtain human body visual features and object visual features.

[0022] S045. Obtain a position embedding corresponding to each query according to the sinusoidal position encoding of the character's prior bounding box.

[0023] S046. Obtain an initial query vector of the query generator of the current layer, and fuse the position embedding to obtain a final query vector of the current layer. The initial query vector of the query generator of the first layer uses the human visual features and the object visual features. The initial query vectors of the query generators of the remaining layers need to be obtained by fusing the feature vector output by the detector of the previous layer with the human visual features and the object visual features.

[0024] In an optional implementation, the calculation model of the final query vector is: .

[0025] .

[0026] In the formula, is the layer number, For the The final query vector of the human body at the layer, For the The final query vector of the layer object, For the The initial query vector of the layer human detector, For the The initial query vector of the layer object detector, For human body position embedding, Embedding for object position, For the The feature vector output by the layer human detector, For the The feature vector output by the layer object detector.

[0027] In an optional embodiment, the human detector and the object detector are both stacked, and the number of stacked layers is Each layer of human detector and each layer of object detector is a Transformer decoder.

[0028] In an optional implementation, the model decoded by the human body detector is: .

[0029] In the formula, For the The feature vector output by the layer human detector, For the Layer human body detector, is the image feature, for the location information, For the The final query vector of the human body at the layer.

[0030] In an optional implementation, the update model for updating the object query vector according to the human body decoding feature is: .

[0031] In the formula, For the The updated object query vector of the layer, For the The final query vector of the layer object, For the The feature vector output by the layer human detector.

[0032] In an optional embodiment, the model decoded by the object detector is: .

[0033] In the formula, For the The feature vector output by the layer object detector, For the Layer object detector, is the image feature, for the location information, For the The updated object query vector for the layer.

[0034] In an optional embodiment, the number of layers of the interactive classifier stack is layers, and each layer of interactive classifiers contains a classification decoder.

[0035] In an optional embodiment, the classification decoder model is: .

[0036] .

[0037] In the formula, is the number of stacked layers, is the layer number, For the Classification decoding of the layer classification decoder output, For the The classification decoder of the layer, is the image feature, for the location information, The character pair decoding features are obtained by fusion and addition of the human decoding features and the object decoding features. The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the Interactive decoding features of layers.

[0038] In an optional implementation, step S08 specifically includes steps S081 to S083.

[0039] S081. Use the human body decoding feature, the object decoding feature, and the initial interaction decoding feature as unary relationship features respectively.

[0040] S082, combining the human body decoding features, the object decoding features and the initial interaction decoding features in pairs in the channel dimension, and then The linear layer performs dimensionality reduction to obtain the pairwise relationship features.

[0041] S083. Combining the human body decoding feature, the object decoding feature and the initial interaction decoding feature as a ternary relationship feature.

[0042] The pairwise relationship feature model is: .

[0043] .

[0044] .

[0045] The ternary relationship feature model is: .

[0046] In the formula, For the The paired relationship features of the human body decoding features and object decoding features of the layer, For the The paired relationship features of the human body decoding features and the initial interaction decoding features of the layer, For the The paired relationship features of the object decoding features of the layer and the initial interaction decoding features, For the linear layer, The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the The initial interactive decoding features output by the layer classification decoder, For the The ternary relationship characteristics of the layer, Represents a merge operation.

[0047] In an optional implementation, step S09 specifically includes steps S091 to S095.

[0048] S091. Obtain the contextual content of the unary relational features through the self-attention mechanism. In the formula, For contextual content, Represents the self-attention mechanism.

[0049] S092, using a cross attention mechanism to embed the context content into the ternary relationship feature to obtain a first embedding feature. In the formula, is the first embedding feature, represents the cross attention mechanism, For the The ternary relationship characteristics of the layer.

[0050] S093. Obtain the paired relationships of paired relationship features through the self-attention mechanism. Among them, In the formula, For paired relationship.

[0051] S094. Use a cross attention mechanism to embed the pairwise relationship into the first embedding feature to obtain a second embedding feature. In the formula, is the second embedded feature.

[0052] S095, using a cross attention mechanism to embed the image feature into the second embedding feature to obtain an enhanced ternary relationship feature. In the formula, For the enhanced ternary relationship features, is the image feature.

[0053] In an optional implementation, step S10 is specifically: The linear layer transforms the enhanced ternary relationship features and the initial interactive decoding features. The transformation model is: .

[0054] .

[0055] In the formula, For the The interaction feature vector of the layer, For the The initial interactive decoding features output by the layer classification decoder, is the conversion factor, Represented as element-wise multiplication, For the enhanced ternary relationship features, for Activation function.

[0056] In an optional implementation, step S02 specifically includes steps S021 to S023.

[0057] S021. Extracting a convolution feature map from the image to be identified through a convolutional neural network module.

[0058] S022. Generate a fixed position code of the same size as the position information of the feature map according to the convolution feature map.

[0059] S023, the position information and the convolution feature Figure 1 The same is input into the Transformer encoder to obtain the encoded image features.

[0060] In an optional implementation, step S03 specifically includes: predicting the feature vector output by the last layer decoder through a head forward network to obtain a final character interaction detection result.

[0061] .

[0062] .

[0063] .

[0064] .

[0065] In the formula, is the human body bounding box, For prediction The forward network, is the human feature vector output by the last layer of human detector, The human body position information updated for the last iteration, is the object bounding box, For prediction The forward network, is the object feature vector output by the last layer of object detector, The object position information updated for the last iteration, For object categories, For prediction The forward network, For the interaction category, For prediction The forward network, is the interactive decoding feature output by the last layer of interactive classifier, for Activation function, for Activation function.

[0066] In the second aspect, the present invention provides a person interaction detection device based on cascade parallelism, which includes an image acquisition module, a feature extraction module, an iterative output module, a query vector module, a human detection module, an object detection module, an initial interaction module, a relationship construction module, an enhancement module and a final interaction module.

[0067] The image acquisition module is used to acquire the image to be recognized.

[0068] The feature extraction module is used to extract image features according to the image to be identified and generate position information of the image features.

[0069] Iteration output module, used for iteration The following steps are performed, and the human interaction detection results are predicted based on the human feature vector, object feature vector and interaction feature vector output by the last iteration.

[0070] The query vector module is used to generate a human query vector and an object query vector according to the image features and the prior bounding box of the person pair, and obtain a query vector group. The prior bounding box of the person pair is obtained by random initialization in the first iteration, and is updated in subsequent iterations according to the human decoding features and object decoding features of the previous iteration.

[0071] The human body detection module is used to obtain human body decoding features through decoding by a human body detector according to the image features, the position information and the human body query vector.

[0072] The object detection module is used to update the object query vector according to the human body decoding feature, and then obtain the object decoding feature by decoding through the object detector according to the updated object query vector, the image feature and the position information.

[0073] The initial interaction module is used to input the image features, the human body decoding features and the object decoding features into a classification decoder to obtain initial interaction decoding features.

[0074] The relationship construction module is used to input the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain unary relationship features, paired relationship features and ternary relationship features.

[0075] The enhancement module is used to embed the unary relationship feature, the pairwise relationship feature and the image feature into the ternary relationship feature for fusion through an attention module.

[0076] The final interaction module is used to enhance the initial interaction decoding features through the fused ternary relationship features to obtain an interaction feature vector.

[0077] In a third aspect, the present invention provides a cascade parallel-based character interaction detection device, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a cascade parallel-based character interaction detection method as described in any paragraph of the first aspect.

[0078] In a fourth aspect, the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for detecting human interaction based on cascaded parallelism as described in any paragraph of the first aspect.

[0079] By adopting the above technical solution, the present invention can achieve the following technical effects: The invention scheme effectively separates the feature learning paths of instance detection and interactive classification tasks through a cascaded parallel decoding architecture, allowing each subtask to be optimized independently and reducing interference between tasks. The architecture combines the feature separation advantages of the two-stage method with the parallel processing efficiency of the one-stage method, significantly reducing the inference time while improving detection accuracy, thus balancing model performance and computing resource consumption.

[0080] In addition, the query optimization generation mechanism provides the decoder with query vectors with clear physical meaning, guiding the model to focus on the target area, reducing interference from irrelevant areas, and further accelerating the model convergence process.

[0081] By introducing the interactive content context learning module, the scheme fully exploits the multiple relationship features between people, objects and interactions, and enhances the semantic expression ability of interactive behaviors. The module integrates global encoding features with local fine-grained context information, strengthens the association reasoning between triples, and effectively improves the classification accuracy in complex interactive scenarios. Experimental results show that the performance indicators of this method on mainstream data sets are better than those of existing technical solutions, verifying its effectiveness and robustness in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 It is a flowchart of a cascade-parallel human interaction detection method.

[0083] Figure 2 It is a schematic diagram of the detection results based on the cascaded parallel human interaction detection method.

[0084] Figure 3 This is a network structure diagram of a cascaded parallel method for detecting human interaction.

[0085] Figure 4 It is the network structure diagram of the feature extractor.

[0086] Figure 5 It is the connection relationship diagram between the query generator, human detector, object detector and interaction classifier.

[0087] Figure 6 It is the network structure diagram of the query builder.

[0088] Figure 7 It is the network structure diagram of the interactive classifier.

[0089] Figure 8 It is a network structure diagram of multiple relationship building blocks. DETAILED DESCRIPTION

[0090] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0091] Example 1, please refer to Figures 1 to 8 , an embodiment of the present invention provides a method for detecting human interaction based on cascade parallelism, such as Figure 3As shown in FIG. 1 , the network structure of the cascade parallel-based human interaction detection method of the present invention mainly includes five modules: feature extractor, query generator, human detector, object detector and interaction classifier. Among them, the query generator, human detector, object detector and interaction classifier are all stacked, and the number of stacked layers is layer. is the sequence number of the layer. Each layer of interactive classifier contains a classification decoder, so the classification decoder is also stacked with layer.

[0092] Each layer of human detector and each layer of object detector is a Transformer decoder, so the human detector is also called a human decoder and the object detector is also called an object decoder.

[0093] The data processing flow of the cascaded parallel-based human interaction detection method is summarized as follows.

[0094] First, for a given image , firstly, the image features are obtained through the feature extractor At the same time, a set of human body prior bounding boxes are randomly initialized and the object prior bounding box , which is collectively referred to as the person pair prior bounding box .

[0095] Then, the query generator, human detector, object detector and interaction classifier iterate to obtain human decoding features, object decoding features and interaction decoding features. , human body prior bounding box and object prior bounding box The query generator generates human query vectors for each layer of human detector and object detector. and the object query vector , and human position embedding and object position embedding Next, the human detector and object detector are based on the input image features. , query vector and its position embedding, generate the corresponding human feature vector and the object feature vector Then, through the human feature vector and the object feature vector , update the prior bounding box, and obtain the decoded interactive feature vector in the interactive classifier. Among them, the interactive feature vector output by the last layer is defined as the interactive decoding feature.

[0096] Finally, the interactive decoding features are processed by the classification head to obtain the prediction results. Among them, the interactive decoding features output by the last layer of interactive classifiers are Used to predict interaction categories, the human feature vector output by the last layer of human detector The feature vector output by the last layer of object detector is used to predict the position of the human body. Used to predict the location and category of objects.

[0097] The following is a detailed introduction to the data processing flow of the cascade parallel-based human interaction detection method. It can be performed by a cascade parallel-based human interaction detection device (hereinafter referred to as: human interaction detection device). In particular, it is performed by one or more processors in the human interaction detection device to implement steps S01 to S10.

[0098] S01. Obtain an image to be recognized.

[0099] Specifically, the image to be identified is used for human-object interaction detection. The concept of human-object interaction (HOI, Human-ObjectInteraction) detection is proposed to solve the problem of detecting the interaction relationship between the human body and objects. HOI detection aims to conduct a deeper analysis of the human-object behavior in the image and learn how people interact with the environment. These interaction relationships reflect human behavior and scene semantics, which is of great significance to improving the intelligence and practicality of computer vision systems.

[0100] It is understandable that the character interaction detection device may be an electronic device with computing capabilities, such as a portable notebook computer, a desktop computer, a server, a smart phone, or a tablet computer.

[0101] S02, extracting image features according to the image to be identified, and generating position information of the image features. In an optional implementation, step S02 specifically includes steps S021 to S023.

[0102] S021. Extracting a convolution feature map from the image to be identified through a convolutional neural network module.

[0103] S022. Generate a fixed position code of the same size as the position information of the feature map according to the convolution feature map.

[0104] S023, the position information and the convolution feature Figure 1 The same is input into the Transformer encoder to obtain the encoded image features.

[0105] Specifically, Figure 4As shown in the figure, the feature extractor consists of a convolutional neural network module and a Transformer encoder to extract image features. , will first pass through the convolutional neural network module to obtain the convolutional feature map . Then convolution feature map Generate fixed position encoding of the same size , as the convolutional feature map The spatial position information of the image is input into the Transformer encoder together with the spatial position information of the image, and the encoded image features are finally obtained. .

[0106] S03, iterate steps S04 to S10 times, and predicts the human interaction detection results based on the human feature vector, object feature vector, and interaction feature vector output by the last iteration. .

[0107] The prior art method of mining the benefits of two-stage and one-stage HOI Detection (CDN, Mining the Benefits of two-stage and One-stage HOI Detection) has some similarities with the embodiments of the present invention.

[0108] Based on CDN, the embodiment of the present invention optimizes the design of query vectors, regards each query in the query vector group as a candidate human interaction detection result, thereby introducing a priori human-object pair bounding boxes and generating query vectors therefrom. By designing query vectors with clear physical meanings, the decoder can focus on specific areas, accelerating the convergence speed of the model.

[0109] Based on CDN, the embodiment of the present invention further optimizes the subtask division of HOI detection and splits a single instance decoder into a human detection decoder and an object detection decoder. The instance detection task is completed by working in parallel with two decoders, and the interactive classification decoder is connected in series. This architecture effectively separates the feature learning paths of target detection and interactive classification.

[0110] In addition, the embodiment of the present invention innovatively proposes an interactive classification decoder to mine the relationship context information between multiple relationships of HOI triples, enhance the expression of interactive features through multi-layer relationship reasoning, and improve the detection accuracy.

[0111] The human body detection decoder, object detection decoder and interactive classification decoder are described in detail below through steps S04 to S10.

[0112] S04. Extracting a human query vector and an object query vector according to the image features and the prior bounding box of the person pair to obtain a query vector group.

[0113] In the first iteration, the character pair prior bounding box is obtained by random initialization. In the subsequent iterations, the character pair prior bounding box is updated according to the human body decoding features and object decoding features of the previous iteration. The character pair prior bounding box includes a human body prior bounding box and an object prior bounding box.

[0114] like Figure 5 and Figure 6 As shown, this embodiment uses a query generator to generate two sets of query vectors for input into each layer of human detector and object detector. and .in, Indicates Individual query vector, Indicates object query vector, is the total number of query vectors.

[0115] like Figure 6 As shown in the figure, the corresponding region of interest (ROIs) features are obtained according to the prior frame, and then the key points of people and objects are predicted from the regional features. Then, the key points are used to sample the visual features, and finally the visual features of the points are fused to generate the human query vector for instance detection. and the object query vector In addition, since the generated query vectors come from the prior bounding boxes, the length of the query vector group depends on the number of prior bounding boxes.

[0116] In an optional implementation, step S04 specifically includes steps S041 to S046.

[0117] S041. According to the human body prior bounding box and the object prior bounding box, obtain human body region features and object region features from the image features, and merge the two region features according to feature dimensions to obtain person pair region features.

[0118] Acquisition of region of interest (ROIs) features based on the human body prior bounding box and the object prior bounding box From the image features Get human body region features ) and object region features , and merge the two regional features according to the feature dimension to obtain the regional features of people-object pairs .

[0119] The calculation model is: .

[0120] .

[0121] .

[0122] In the formula, Regional characteristics of the human body, For the The human body region features corresponding to the query vectors, Object area characteristics, For the The object region features corresponding to the query vector, is the total number of query vectors, For the regional characteristics of people-things, is the image feature, is the human body prior bounding box, is the object prior bounding box, It represents the sampling operation of obtaining regional features from the encoded image features. It is a merge operation.

[0123] S042, according to the regional features of the characters, through the convolution layer plus the multi-layer perceptron The activation functions predict the key points of the human body and the key points of the object respectively.

[0124] About the prediction of key points. For each regional feature and , through the convolution layer (Conv) and the multi-layer perceptron (MLP) respectively for the key points of the human body and object key points Make predictions. for No. Key points, for No. Key points, is the total number of key points.

[0125] Considering that the coordinates of the key points need to be normalized to [-1,1] during feature sampling, the prediction value of the key points is output using The activation function undergoes nonlinear changes to ensure the output coordinate range.

[0126] The prediction model of the key points is: .

[0127] .

[0128] in, represents the convolution operation, represents a multilayer perceptron (MLP), express Activation function. In this embodiment, the MLP consists of three fully connected layers to map the input features to two-dimensional point information.

[0129] S043. According to the key points of the human body and the key points of the object, a bilinear interpolation sampling operation is performed on the regional features of the human body to extract local visual features of the human body and local visual features of the object respectively.

[0130] For each instance, multiple key points are predicted, and the features sampled from these key points are fused to construct the visual features of the instance. The key point feature sampling process includes using the key points of the human body and object key points Regional characteristics of people-object pairs Bilinear interpolation sampling is performed to extract the local visual features of the human body and the local visual features of the object. In this way, the local information of each key point can be effectively obtained from the regional features.

[0131] The model for extracting local visual features is as follows: ; ; In the formula, The local visual features of the human body, is the local visual feature of the object, Represents a sampling operation.

[0132] S044. Aggregate the local visual features of the human body and the local visual features of the object through a fully connected layer to obtain human body visual features and object visual features.

[0133] After extracting local visual features, the local visual features in the same instance are aggregated through a fully connected layer to obtain the visual features of the human body and the visual features of the object.

[0134] .

[0135] .

[0136] In the formula, represents a fully connected layer.

[0137] S045. Obtain a position embedding corresponding to each query according to the sinusoidal position encoding of the character's prior bounding box.

[0138] Specifically, the sinusoidal position encoding of the prior box is used as the position embedding corresponding to each query.

[0139] The calculation model of position embedding is: .

[0140] .

[0141] In the formula, For human body position embedding, Embedding for object position, represents the sine embedding function, is the human body prior bounding box, is the object prior bounding box. The sine embedding function is used to convert point coordinates into feature vectors, which is the same as that used in target detection.

[0142] S046. Obtain an initial query vector of the query generator of the current layer, and fuse the position embedding to obtain a final query vector of the current layer. The initial query vector of the query generator of the first layer uses the human visual features and the object visual features. The initial query vectors of the query generators of the remaining layers need to be obtained by fusing the feature vector output by the detector of the previous layer with the human visual features and the object visual features.

[0143] Since the human detector and object detector are composed of stacked decoders, the corresponding query generators are also stacked.

[0144] In addition to the query generator in the first layer, the visual features of human bodies and objects are directly used. In addition to the query vectors for human detectors and object detectors, the queries of each layer need to fuse the decoded feature vectors of the previous layer to generate the final query vector of the current layer. , thereby retaining the effective information of the previous layer. Indicates Final query vector for each person. Indicates The final query vector for each object.

[0145] Preferably, the calculation model of the final query vector is: .

[0146] .

[0147] In the formula, is the layer number, For the The final query vector of the human body at the layer, For the The final query vector of the layer object, For the The initial query vector of the layer human detector, For the The initial query vector of the layer object detector, For human body position embedding, Embedding for object position, For the The feature vector output by the layer human detector, For the The feature vector output by the layer object detector.

[0148] The initial query vector of the current layer is

[0149]

[0150] In the formula, and Expressed as Initial query vectors for layer human and object detectors.

[0151] Specifically, both the human detector and the object detector are composed of stacked Transformer decoders. The query vector group, image features, and corresponding position information generated by the query optimization generator are input into the corresponding decoders, and the decoders output the decoded feature vectors. The bounding box information is updated by the human decoded feature vector and the object decoded feature vector, and is input into the interactive context content learning module for interactive feature learning.

[0152] S05. According to the image features, the position information and the human query vector, obtain human decoding features through decoding by a human detector.

[0153] Preferably, the model decoded by the human body detector is: .

[0154] In the formula, For the The feature vector output by the layer human detector, For the Layer human body detector, is the image feature, for the location information, For the The final query vector of the human body at the layer.

[0155] S06. Update the object query vector according to the human body decoding feature, and then obtain the object decoding feature by decoding through the object detector according to the updated object query vector, the image feature and the position information.

[0156] The update model for updating the object query vector according to the human body decoding feature is: .

[0157] The model for decoding via the object detector is: .

[0158] In the formula, For the The updated object query vector of the layer, For the The final query vector of the layer object, For the The feature vector output by the layer human detector, For the The feature vector output by the layer object detector, For the Layer object detector, is the image feature, for the location information, For the The updated object query vector for the layer.

[0159] Specifically, after each layer of human detector and object detector outputs a feature vector, it will be used to update the bounding box of the prior human-object pair: .

[0160] .

[0161] In the formula, For the The human body prior bounding box of the layer, For the The feature vector output by the layer human detector, For the The human body prior bounding box of the layer, For the The object prior bounding box of the layer, For the The feature vector output by the layer object detector, For the The object prior bounding box of the layer, for Activation function, For prediction The forward network, For prediction The forward network.

[0162] Considering the key role of the relationship context between people, objects and interactions in HOI instance detection, this embodiment designs an interaction content context learning module (i.e., interaction classification module). This module mines multiple relationship contexts in triples to enhance the expressiveness of interaction features. Multiple relationships include unary, paired and ternary relationship contexts, where unary relationships are people, objects, and interaction features, paired relationships are people-objects, people-interactions, and objects-interaction features, and ternary relationships cover people-objects-interaction features.

[0163] Unary and pairwise relations focus on task understanding at a fine-grained level and provide fine-grained contextual information of HOI instances. Triple relations provide contextual information of the entire person, thereby understanding the HOI instance at a holistic level. In order to make full use of this holistic and fine-grained contextual information, unary and pairwise relations are embedded in the context of triple relations to effectively generate enhanced interaction features. Its specific structure is as follows: Figure 7 shown.

[0164] First, the decoded features and image encoding features output by the j-th layer human detector and object detector are input into the classification decoder module to obtain the initial interactive decoding features (i.e., step S07). Then, the human decoding features, object decoding features, and initial interactive decoding features are input into the multiple relationship construction module to generate unary relationship features, pairwise relationship features, and ternary relationship features (i.e., steps S08 and S09), as shown in Figure 7 shown.

[0165] Next, the context information of the unary relation is extracted through the self-attention mechanism, and the unary relation context is embedded into the ternary relation feature using the cross-attention mechanism. Similarly, the context content of the pairwise relation is extracted using the self-attention mechanism and embedded into the ternary relation context through the cross-attention. In addition, the global image encoding feature is also embedded into the ternary relation context through the cross-attention calculation. Finally, by utilizing the enhanced ternary relation context information, the expressive power of the interactive feature is enhanced.

[0166] S07: Input the image features, the human body decoding features and the object decoding features into a classification decoder to obtain initial interactive decoding features.

[0167] Preferably, the classification decoder model is: .

[0168] .

[0169] In the formula, is the number of stacked layers, is the layer number, For the Classification decoding of the layer classification decoder output, For the The classification decoder of the layer, is the image feature, for the location information, The character pair decoding features are obtained by fusion and addition of the human decoding features and the object decoding features. The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the Interactive decoding features of layers.

[0170] S08. Input the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain unary relationship features, paired relationship features and ternary relationship features.

[0171] Preferably, merging is performed in the channel dimension, and the dimension is reduced from a high dimension to a low dimension through MLP (multiple linear layers). Step S08 specifically includes steps S081 to S083.

[0172] S081, using the human body decoding feature, the object decoding feature and the initial interaction decoding feature as unary relationship features respectively. Figure 8 shown.

[0173] S082, combining the human body decoding features, the object decoding features and the initial interaction decoding features in pairs in the channel dimension, and then The linear layer performs dimensionality reduction to obtain the pairwise relationship features.

[0174] Preferably, the pairwise relationship feature model is: .

[0175] .

[0176] .

[0177] In the formula, For the The paired relationship features of the human body decoding features and object decoding features of the layer, For the The paired relationship features of the human body decoding features and the initial interaction decoding features of the layer, For the The paired relationship features of the object decoding features of the layer and the initial interaction decoding features, For the linear layer, The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the Initial interactive decoded features output by the layer classification decoder.

[0178] S083. Combining the human body decoding feature, the object decoding feature and the initial interaction decoding feature as a ternary relationship feature.

[0179] Preferably, the ternary relationship feature model is: .

[0180] In the formula, For the The ternary relationship characteristics of the layer, For the linear layer, Indicates a merge operation. The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the Initial interactive decoded features output by the layer classification decoder.

[0181] S09. Embed the unary relationship feature, the paired relationship feature and the image feature into the ternary relationship feature through an attention module for fusion.

[0182] In an optional embodiment, if Figure 7 As shown, step S09 specifically includes steps S091 to S095.

[0183] S091. Obtain the contextual content of unary relation features through self-attention mechanism.

[0184] .

[0185] In the formula, For contextual content, Represents the self-attention mechanism.

[0186] S092. Use a cross-attention mechanism to embed the context content into the ternary relationship feature to obtain a first embedded feature.

[0187] .

[0188] In the formula, is the first embedding feature, represents the cross attention mechanism, For the The ternary relationship characteristics of the layer.

[0189] S093. Obtain pairwise relationships of pairwise relationship features through self-attention mechanism.

[0190] .

[0191] In the formula, For paired relationship.

[0192] S094. Use a cross-attention mechanism to embed the pairwise relationship into the first embedding feature to obtain a second embedding feature.

[0193] .

[0194] In the formula, is the second embedded feature.

[0195] S095. Use a cross-attention mechanism to embed the image feature into the second embedded feature to obtain an enhanced ternary relationship feature.

[0196] .

[0197] In the formula, For the enhanced ternary relationship features, is the image feature.

[0198] S10. Enhance the initial interactive decoding features by using the fused ternary relationship features to obtain an interactive feature vector.

[0199] Preferably, step S10 is specifically: The linear layer transforms the enhanced ternary relationship features and the initial interaction decoding features, and selects necessary context information for them to generate rich interaction features.

[0200] The conversion model is: .

[0201] .

[0202] In the formula, For the The interaction feature vector of the layer, For the The initial interactive decoding features output by the layer classification decoder, is the conversion factor, Represented as element-wise multiplication, For the enhanced ternary relationship features, for Activation function.

[0203] Based on the above embodiments, in an optional embodiment of the present invention, as Figure 3 As shown, when iterating step S04 to step S10 After that, step S03 is specifically as follows: according to the feature vector output by the last layer decoder, prediction is performed through the head forward network to obtain the final character interaction detection result.

[0204] The prediction model based on the forward network is: .

[0205] .

[0206] .

[0207] .

[0208] In the formula, for Activation function, for Activation function, is the human body bounding box, For prediction The forward network, is the human feature vector output by the last layer of human detector, The human body position information updated for the last iteration, is the object bounding box, For prediction The forward network, is the object feature vector output by the last layer of object detector, The object position information updated for the last iteration, For object categories, For prediction The forward network, For the interaction category, For prediction The forward network, The interactive decoding features output by the last layer of interactive classifier.

[0209] Specifically, the last layer output , , It will pass through the head forward network for prediction to get the bounding box containing the human body , object bounding box , object category and interaction categories The prediction results of character interaction.

[0210] The cascaded parallel-based character interaction detection method of the present invention effectively solves some problems existing in the character interaction detection method based on the encoder-decoder structure, and realizes efficient character interaction behavior detection. It has the following advantages.

[0211] 1. Design a query optimization generator to generate query vectors related to the target area, guide the decoder to focus on specific areas of the image, and reduce the interference of irrelevant areas on the decoder, thereby improving the training efficiency of the model and accelerating the convergence process. At the same time, a forward guided query mechanism is used for the query vector of each subtask, which enhances the correlation between the three subtask branches of HOI, thereby promoting information flow and mutual collaboration between tasks.

[0212] 2. Design a cascaded parallel decoding Transformer architecture to effectively separate the learning of instance features and interaction features. This architecture allows each subtask to focus on the current stage and speed up reasoning, making full use of the advantages of the two-stage method based on Transformer and the one-stage method.

[0213] 3. Design an interactive context content learning module to extract overall and fine-grained context information by mining multiple relational contexts in triples, effectively enhancing the expressiveness of interactive features. This module effectively strengthens the connection between different tasks through rich context reasoning, promotes better integration of interactive information, and improves the accuracy of interactive action classification.

[0214] Embodiment 2: The embodiment of the present invention provides a person interaction detection device based on cascade parallelism, which includes an image acquisition module, a feature extraction module, an iterative output module, a query vector module, a human detection module, an object detection module, an initial interaction module, a relationship construction module, an enhancement module and a final interaction module.

[0215] The image acquisition module is used to acquire the image to be recognized.

[0216] The feature extraction module is used to extract image features according to the image to be identified and generate position information of the image features.

[0217] Iteration output module, used for iteration The following steps are performed, and the human interaction detection results are predicted based on the human feature vector, object feature vector and interaction feature vector output by the last iteration.

[0218] The query vector module is used to generate a human query vector and an object query vector according to the image features and the prior bounding box of the person pair, and obtain a query vector group. The prior bounding box of the person pair is obtained by random initialization in the first iteration, and is updated in subsequent iterations according to the human decoding features and object decoding features of the previous iteration.

[0219] The human body detection module is used to obtain human body decoding features through decoding by a human body detector according to the image features, the position information and the human body query vector.

[0220] The object detection module is used to update the object query vector according to the human body decoding feature, and then obtain the object decoding feature by decoding through the object detector according to the updated object query vector, the image feature and the position information.

[0221] The initial interaction module is used to input the image features, the human body decoding features and the object decoding features into a classification decoder to obtain initial interaction decoding features.

[0222] The relationship construction module is used to input the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain unary relationship features, paired relationship features and ternary relationship features.

[0223] The enhancement module is used to embed the unary relationship feature, the pairwise relationship feature and the image feature into the ternary relationship feature for fusion through an attention module.

[0224] The final interaction module is used to enhance the initial interaction decoding features through the fused ternary relationship features to obtain an interaction feature vector.

[0225] Embodiment 3: The embodiment of the present invention provides a cascade parallel-based character interaction detection device, which includes a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a cascade parallel-based character interaction detection method as described in any paragraph of Embodiment 1.

[0226] Embodiment 4: Embodiments of the present invention provide a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute a method for detecting human interaction based on cascaded parallelism as described in any paragraph of Embodiment 1.

[0227] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0228] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0229] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, electronic device, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk. It should be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0230] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0231] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0232] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0233] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0234] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting human interaction based on cascade parallelism, characterized in that: Include: Obtain an image to be recognized; Extracting image features according to the image to be identified, and generating position information of the image features; Iteration The following steps are performed, and the human interaction detection result is predicted based on the human feature vector, object feature vector and interaction feature vector outputted in the last iteration; Generate a human query vector and an object query vector according to the image features and the prior bounding box of the person pair, and obtain a query vector group; wherein the prior bounding box of the person pair is obtained by random initialization in the first iteration, and is updated according to the human decoding features and object decoding features of the previous iteration in subsequent iterations; Obtaining a human body decoding feature by decoding through a human body detector according to the image feature, the position information and the human body query vector; updating the object query vector according to the human body decoding feature, and then decoding the object decoding feature by an object detector according to the updated object query vector, the image feature and the position information; Inputting the image features, the human body decoding features and the object decoding features into a classification decoder to obtain initial interactive decoding features; Inputting the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain unary relationship features, paired relationship features and ternary relationship features; The unary relationship feature, the pairwise relationship feature and the image feature are embedded into the ternary relationship feature for fusion through an attention module; The initial interactive decoding features are enhanced by the fused ternary relationship features to obtain an interactive feature vector.

2. The method for detecting human interaction based on cascaded parallelism according to claim 1, characterized in that: The character pair prior bounding box includes a human body prior bounding box and an object prior bounding box; The query generator, human detector, and object detector are all stacked, and the number of stacking layers is layer; According to the image features and the prior bounding box of the person, a human query vector and an object query vector are generated to obtain a query vector group, which specifically includes: According to the human body prior bounding box and the object prior bounding box, obtaining human body region features and object region features from the image features, and merging the two region features according to feature dimensions to obtain human body region features; According to the regional features of the characters, the convolutional layer plus the multi-layer perceptron is added The activation function predicts the key points of the human body and the key points of the object respectively; According to the key points of the human body and the key points of the object, a bilinear interpolation sampling operation is performed on the regional features of the human body to extract local visual features of the human body and local visual features of the object respectively; Aggregating the local visual features of the human body and the local visual features of the object through a fully connected layer to obtain the visual features of the human body and the visual features of the object; Obtaining a position embedding corresponding to each query based on the sinusoidal position encoding of the person's prior bounding box; Obtain an initial query vector of the query generator of the current layer, and fuse the position embedding to obtain a final query vector of the current layer; wherein the initial query vector of the query generator of the first layer uses the human visual features and the object visual features; the initial query vectors of the query generators of the remaining layers need to be obtained by fusing the feature vector output by the detector of the previous layer with the human visual features and the object visual features.

3. The method for detecting human interaction based on cascaded parallelism according to claim 2, characterized in that: The calculation model of the final query vector is: ； ； In the formula, is the layer number, For the The final query vector of the human body at the layer, For the The final query vector of the layer object, For the The initial query vector of the layer human detector, For the The initial query vector of the layer object detector, For human body position embedding, Embedding for object position, For the The feature vector output by the layer human detector, For the The feature vector output by the layer object detector.

4. The method for detecting human interaction based on cascaded parallelism according to claim 1, characterized in that: Both human detectors and object detectors are stacked, and the number of stacked layers is Layer; each layer of human detector and each layer of object detector is a Transformer decoder; The model decoded by the human detector is: ； In the formula, For the The feature vector output by the layer human detector, For the Layer human body detector, is the image feature, for the location information, For the The final query vector of the human body at the layer; The update model for updating the object query vector according to the human body decoding feature is: ； In the formula, For the The updated object query vector of the layer, For the The final query vector of the layer object, For the The feature vector output by the layer human detector; The model for decoding via the object detector is: ； In the formula, For the The feature vector output by the layer object detector, For the Layer object detector, is the image feature, for the location information, For the The updated object query vector for the layer.

5. The method for detecting human interaction based on cascaded parallelism according to claim 1, characterized in that: The number of layers of the interactive classifier stack is layers, and each layer of interactive classifiers contains a classification decoder; The classification decoder model is: ； ； In the formula, is the number of stacked layers, is the layer number, For the Classification decoding of the layer classification decoder output, For the The classification decoder of the layer, is the image feature, for the location information, The character pair decoding features are obtained by fusion and addition of the human decoding features and the object decoding features. The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the Interactive decoding features of layers.

6. The method for detecting human interaction based on cascaded parallelism according to claim 1, characterized in that: Inputting the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain unary relationship features, paired relationship features and ternary relationship features, specifically including: The human body decoding feature, the object decoding feature and the initial interaction decoding feature are respectively used as unary relationship features; The human body decoding features, the object decoding features and the initial interaction decoding features are combined in pairs in the channel dimension, and then The linear layer performs dimensionality reduction to obtain the pairwise relationship features; Combining the human body decoding feature, the object decoding feature and the initial interaction decoding feature as a ternary relationship feature; The pairwise relationship feature model is: ； ； ； The ternary relationship feature model is: ； In the formula, For the The paired relationship features of the human body decoding features and object decoding features of the layer, For the The paired relationship features of the human body decoding features and the initial interaction decoding features of the layer, For the The paired relationship features of the object decoding features of the layer and the initial interaction decoding features, For the linear layer, The human body detector The feature vector output by the layer decoder, The object detector The feature vector output by the layer decoder, For the The initial interactive decoding features output by the layer classification decoder, For the The ternary relationship characteristics of the layer, Indicates a merge operation; The unary relationship feature, the paired relationship feature and the image feature are embedded into the ternary relationship feature through an attention module for enhancement, specifically including: The contextual content of the unary relation feature is obtained through the self-attention mechanism; ; In the formula, For contextual content, Represents the self-attention mechanism; The context content is embedded into the ternary relationship feature using a cross attention mechanism to obtain a first embedding feature; wherein, ; In the formula, is the first embedding feature, represents the cross attention mechanism, For the The ternary relationship characteristics of the layers; The pairwise relationships of pairwise relationship features are obtained through the self-attention mechanism; among them, ; In the formula, For paired relationships; The pairwise relationship is embedded into the first embedding feature using a cross attention mechanism to obtain a second embedding feature; wherein, ; In the formula, is the second embedded feature; The image feature is embedded into the second embedding feature using a cross attention mechanism to obtain an enhanced ternary relationship feature; wherein, ; In the formula, For the enhanced ternary relationship features, is the image feature; The enhanced ternary relationship feature and the initial interactive decoding feature are converted to obtain the interactive decoding feature, specifically including: pass The linear layer transforms the enhanced ternary relationship features and the initial interactive decoding features; the transformation model is: ； ； In the formula, For the The interaction feature vector of the layer, For the The initial interactive decoding features output by the layer classification decoder, is the conversion factor, Represented as element-wise multiplication, For the enhanced ternary relationship features, for Activation function.

7. A cascaded parallel-based person interaction detection method according to any one of claims 1 to 6, characterized in that: Extracting image features according to the image to be identified and generating location information of the image features specifically includes: Extracting a convolution feature map from the image to be identified by a convolutional neural network module; Generate a fixed position code of the same size as the position information of the feature map according to the convolution feature map; Inputting the position information and the convolution feature map into a Transformer encoder to obtain encoded image features; Iteration The following steps are performed, and the last output is used as the final human interaction detection result, including: According to the feature vector output by the last layer decoder, prediction is performed through the head forward network to obtain the final human interaction detection result; ； ； ； ； In the formula, is the human body bounding box, For prediction The forward network, is the human feature vector output by the last layer of human detector, The human body position information updated for the last iteration, is the object bounding box, For prediction The forward network, is the object feature vector output by the last layer of object detector, The object position information updated for the last iteration, For object categories, For prediction The forward network, For the interaction category, For prediction The forward network, is the interactive decoding feature output by the last layer of interactive classifier, for Activation function, for Activation function.

8. A cascade-based parallel human interaction detection device, characterized in that: Include: An image acquisition module, used for acquiring an image to be identified; A feature extraction module, used to extract image features according to the image to be identified, and generate position information of the image features; Iteration output module, used for iteration The following steps are performed, and the human interaction detection result is predicted based on the human feature vector, object feature vector and interaction feature vector outputted in the last iteration; A query vector module, used to generate a human query vector and an object query vector according to the image features and the prior bounding box of the person pair, so as to obtain a query vector group; wherein the prior bounding box of the person pair is obtained by random initialization in the first iteration, and is updated according to the human decoding features and object decoding features of the previous iteration in subsequent iterations; A human body detection module, configured to obtain human body decoding features by decoding through a human body detector according to the image features, the position information and the human body query vector; An object detection module, used for updating the object query vector according to the human body decoding feature, and then obtaining the object decoding feature by decoding through an object detector according to the updated object query vector, the image feature and the position information; An initial interaction module, used for inputting the image feature, the human body decoding feature and the object decoding feature into a classification decoder to obtain an initial interaction decoding feature; A relationship construction module, used for inputting the human body decoding feature, the object decoding feature and the initial interaction decoding feature into a multiple relationship construction module to obtain a unary relationship feature, a pairwise relationship feature and a ternary relationship feature; An enhancement module, used for embedding the unary relationship feature, the paired relationship feature and the image feature into the ternary relationship feature for fusion through an attention module; The final interaction module is used to enhance the initial interaction decoding features through the fused ternary relationship features to obtain an interaction feature vector.

9. A cascade-based parallel human interaction detection device, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory; the computer program can be executed by the processor to implement a cascaded parallel-based human interaction detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a cascade-parallel-based character interaction detection method as described in any one of claims 1 to 7.

Images