A semantic human interaction detection method based on a cascaded decoder

Abstract

The application discloses a semantic person interaction detection method based on a cascade decoder, wherein a to-be-detected image is input into a backbone network to extract features, the extracted features are flattened into one-dimensional vectors as image visual features for subsequent processing, then the image visual features are input into a shared encoder to obtain coding features, after that, a person-object decoder and a verb decoder are adopted for decoding, a semantic branch is adopted to obtain a confidence score, the confidence score is adopted to constrain output, and finally, a final detection result is obtained. The application uses the structure of the cascade decoder to better obtain the features of the person-object and the action, and uses a pre-trained semantic module to constrain the output of the prediction result, finally outputs the interaction result, and the precision of the person-object interaction detection is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of human-object interaction detection, and particularly relates to a human-object interaction detection method based on a cascade decoder. BACKGROUND

[0002] Human-object interaction detection is a downstream task of target detection. In the basic visual task of target detection, given an image, all target sets in the image are detected, and the target set is represented as where N represents the number of targets, and represents outputting it in the form of a target frame and a target category, wherein, is a target frame, is a category. In the human-object interaction detection task, the task pays more attention to the relationship interaction between people and objects. Given an input image, the triplet in the form of <human, object, interaction> is required to be inferred, which is represented as Triplet i =< h i ,o i ,v i >,h i ={human bbox i},oi={object bbox i ,object category i},v i = {verb category i},wherein Triplet i represents the predicted triplet result, and respectively represents the prediction result of a related person, the prediction result of a related object, and the prediction result of a human-object action interaction. h i represents the prediction result related to a person, human bbox is a person bounding box. o i represents the prediction result related to an object, object bbox i is an object bounding box, object category is an object category. v i is the prediction result of a human-object action interaction, verb category i is an action category. Compared with target detection, human-object interaction detection pays more attention to the understanding of the interaction scene related to a person.

[0003] There are several typical human-object interaction detection methods at present. First, the two-stage human-object interaction detection algorithm, which usually uses R-CNN to locate the relevant area, and uses the backbone network to crop the features, and finally uses the multi-branch flow method to process the cropped features. Each branch flow corresponds to different features, and finally fuses the features to output the results. However, using the cropped features to predict the action category may cause the corresponding context information to be polluted by irrelevant features, resulting in performance degradation. Second, the single-stage human-object relationship detection method uses the comprehensive features of a pair of people and an object to detect, and uses the CenterNet network as the backbone network, and completes the interaction detection based on points. However, these methods still have limitations, that is, the traditional convolutional network cannot introduce global information, which will also cause feature pollution. Third, the end-to-end DETR method has become popular, which uses deep self-attention instead of convolution, and can introduce global information to complete set prediction. It is a very natural idea to introduce DETR into various fields of vision, including using DETR to complete instance segmentation and human-object relationship detection. The QPIC algorithm introduces DETR into the human-object relationship interaction detection field, uses it as a basic detector, and extracts the corresponding context information to predict the final triple set. However, the related human-object interaction detection algorithm based on DETR directly completes the triple set prediction, which predicts <human, object, action> (the triple prediction of human, object and action) as a whole set, ignoring the feature relationship between human and action, and object and action. SUMMARY

[0004] The purpose of the present application is to provide a semantic human interaction detection method based on a cascading decoder to overcome the problem of ignoring feature relationships and lacking reliability judgment in the prior art.

[0005] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0006] A semantic human interaction detection method based on a cascading decoder, which is based on a human-object relationship interaction detection network for detection, the human-object relationship interaction detection network includes a backbone network, a shared encoder, a human-object decoder, a verb decoder and a semantic branch, the semantic branch includes a triple encoder, a sigmoid function and a confidence encoding layer, the semantic human interaction detection method based on the cascading decoder, comprising:

[0007] The image to be detected is input into the backbone network to extract features, and the extracted features are flattened into a one-dimensional vector as the image visual features for subsequent processing;

[0008] input the image visual features into the shared encoder to obtain human-object encoding features, shared encoding features and visual encoding features;

[0009] input the human-object encoding features into the human-object decoder to obtain human-object decoding features, and then pass the human-object decoding features through a fully connected layer to obtain the bounding box results of the human and the object and the category of the object;

[0010] input the shared encoding features and the human-object decoding features into the verb decoder to obtain verb category features, and pass the verb category features through a fully connected layer to obtain the verb classification result;

[0011] obtain valid triple data of the image to be detected, input the valid triple data into a triple encoder to obtain triple encoding features, and then use the visual encoding features and the triple encoding features to construct a semantic confidence matrix, perform a sigmoid function operation on the semantic confidence matrix, and pass the semantic confidence matrix through a confidence encoding layer to obtain corresponding confidence scores;

[0012] use the bounding box results of the human and the object, the category of the object and the verb classification result as predicted triples, use the confidence scores to constrain the predicted triples, and obtain the final detection result.

[0013] Further, the shared encoder and the triple encoder both adopt a self-attention mechanism.

[0014] Further, the human-object decoder and the verb decoder both adopt a deep self-attention transformer network.

[0015] Further, the obtaining of the valid triple data of the image to be detected comprises:

[0016] inputting the image to be detected into a pre-trained CLIP model to obtain the valid triple data.

[0017] Further, the use of the bounding box results of the human and the object, the category of the object and the verb classification result as predicted triples, the use of the confidence scores to constrain the predicted triples, and the obtaining of the final detection result comprise:

[0018] if there is at least one predicted triple with a confidence score greater than a confidence threshold, output the predicted triple with the confidence score greater than the confidence threshold as the detection result;

[0019] if the confidence scores of all predicted triples are not greater than the confidence threshold, select the predicted triple with the highest confidence score as the detection result.

[0020] Further, the overall loss function of the human-object relationship interaction detection network is composed of two parts, the first part is a detection loss, and the second part is a semantic loss:

[0021]

[0022] wherein the detection loss is represented as follows:

[0023]

[0024] wherein, the detection loss is composed of four parts, represents the bounding box regression loss, λ u represents the interaction joint Iou loss, represents the classification loss, represents the verb loss, λ b , λ u , λ c , λ a are weight parameters respectively.

[0025] The semantic human interaction detection method of the cascaded decoder proposed in the present application uses a deep self-attention transformation network to complete the encoding and decoding operation on visual features, and obtains the acquisition of global visual features. In order to enhance the feature connection between human-action and object-action, a cascaded decoder structure is proposed to better fuse features, and a pre-trained semantic module is proposed to constrain the output, and a loss function is constructed according to the characteristics of the semantic module, to complete the gradient training of the module and improve the prediction accuracy of the interaction result. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flow chart of the semantic human interaction detection method based on the cascaded decoder of the present application;

[0027] Figure 2 is a structural diagram of the human-object relationship interaction detection network of the present application;

[0028] Figure 3 is a structural diagram of the semantic branch. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0030] As Figure 1 shown, a semantic human interaction detection method based on a cascaded decoder is provided, which is detected based on a constructed human-object relationship interaction detection network, comprising:

[0031] Step S1, input the image to be detected into the backbone network to extract features, and flatten the extracted features into a one-dimensional vector as the image visual features for subsequent processing.

[0032] As shown in Figure 2 The human-object relationship interaction detection network includes a backbone network, a shared encoder, a human-object decoder, a verb decoder, and a semantic branch including a triplet encoder, a sigmoid function, and a confidence embedding layer.

[0033] The application adopts the convolutional backbone network of ResNet50 or ResNet101 to extract preliminary image visual features.

[0034] When training the network model, the method of data augmentation is usually adopted to pre-process the collected images to obtain the processed image data as the training data set. The specific pre-processing can include random flipping of the image, random cropping, scaling the image size to a uniform size and normalizing the image, etc., and finally inputting the image into the network model in batches to complete the training of the network model. In actual application, for the image to be detected, only scaling to a uniform size and normalization are required.

[0035] The pre-processed image to be detected is input into the backbone network, and the output is the extracted feature map. In this embodiment, the extracted feature map is flattened into a one-dimensional vector as the image visual features for subsequent processing.

[0036] Step S2, input the image visual features into the shared encoder to obtain human-object encoding features, shared encoding features, and visual encoding features.

[0037] As shown in Figure 2 In this step, the image visual features are input into the shared encoder. In this embodiment, the shared encoder uses a self-attention mechanism to obtain the encoding features. The self-attention mechanism is a relatively mature technology in the field, and will not be described here. Finally, after passing through the shared encoder, the encoding features output are obtained: human-object encoding features (human object memory), shared encoding features (shared memory), and visual encoding features (visual memory). In this embodiment, the human-object encoding features, the shared encoding features, and the visual encoding features are all the same encoding features output by the shared encoder.

[0038] Step S3, input the human-object encoding features into a human-object decoder to obtain human-object decoding features, and then pass the human-object decoding features through a fully connected layer to obtain the bounding box results of the human and the object and the category of the object.

[0039] After obtaining the human-object encoding features, the embodiment inputs the human-object encoding features into a human-object decoder. The human-object decoder adopts a decoder of a deep self-attention transformer. The deep self-attention transformer is a mature technology in the field, and thus will not be described herein.

[0040] First, an initialization feature map human object queries with a length of 100 is initialized as the input of the human-object decoder. The initialization manner is random initialization with a dimension of Nxd, wherein d represents the channel number and N represents the length. In the embodiment, N is defined as 100, and d is usually taken as 256 in the human-object detection task. The feature map is used to describe human-object related feature information and provide context data information for subsequent processes.

[0041] Then, the initialized feature map and the human-object encoding features are input into the human-object decoder to obtain decoded human-object set features, i.e., human-object decoding features.

[0042] Then, the human-object decoding features are input into a fully connected layer FFN to obtain the position of the human and the object and the category of the object. The formula is as follows:

[0043] Human bbox = Sigmoid(FFN sub bboxembed (Feature humanobject ))

[0044] Object bbox = Sigmoid(FFN obj bboxembed (Feature humaobject ))

[0045] Object class = FFN obj classembed (Feature humaobject )

[0046] wherein Feature humaobject represents the human-object set decoding features, FFN bboxembed corresponds to a fully connected layer for different inputs, and finally the bounding box results of the human and the object and the category of the object are obtained.

[0047] Step S4, input the shared encoding features and the human-object decoding features into a verb decoder to obtain verb category features, and pass through a fully connected layer to obtain a verb classification result.

[0048] This step inputs the shared encoding features and the human-object decoding features into a verb decoder. The verb decoder also adopts a decoder of a deep self-attention transformer.

[0049] The verb decoder of this embodiment does not need to be initialized, and the features can be directly input. Since the features contain the collective features of the human and the object, the use of this structure can better explore the feature relationship between the human, the object and the action. Verb category features are obtained, and finally input to a fully connected layer FFN to obtain a verb classification result. The formula is as follows:

[0050] Verb class =FFN verb classembed (Feature verb )

[0051] Where Feature verb represents the verb category features, and Verb class represents the verb classification result.

[0052] It should be noted that the human-object decoder and the verb decoder in this embodiment both adopt a decoder of a deep self-attention transformer. The decoder needs two inputs, one is an encoding feature, and the other is an initialization feature map. The input of the human-object decoder is the initialization feature map and the human-object encoding feature; the input of the verb decoder is the human-object decoding feature obtained from the human-object decoder and the shared encoding feature obtained from the shared encoder, wherein the human-object decoding feature is used as the initialization feature map of the verb decoder.

[0053] Step S5, obtain valid triple data of the to-be-detected image, input the triple data to a triple encoding encoder to obtain triple encoding features, then construct a semantic confidence matrix by using the visual encoding features and the triple encoding features, perform a sigmoid function operation on the semantic confidence matrix, pass through a confidence encoding layer, and obtain a corresponding confidence score.

[0054] Specifically, Figure 3As shown, this step actually describes the semantic branch of the entire network, and the semantic module of the semantic branch includes a triplet encoder. First, the image to be detected is input into a pre-trained CLIP model to obtain effective triplet data. The CLIP model is a network model that identifies a picture as interactive text, and its prediction result is a text sequence of human-object-action. In actual detection, the pre-trained CLIP model will output a description text for the image to be detected, for example: the photo of human playing basketball, as effective triplet data.

[0055] Subsequently, the effective triplet data is input into a triplet encoder (Triplet Encoder), which has the same structure as the shared encoder, to obtain triplet memory.

[0056] The semantic confidence matrix is constructed using the triplet memory and visual memory (visual memory). The semantic confidence matrix can be specifically represented as:

[0057] semantic matrix = triplet memory T * visual memory

[0058] Wherein, the triplet memory is the triplet memory, the visual memory is the visual memory obtained from the shared encoder, and they are simply multiplied element by element to obtain the semantic confidence matrix.

[0059] Finally, the sigmoid function is accumulated and passed through the confidence embedding layer to obtain the final confidence score. The formula is as follows:

[0060]

[0061] The confidence embedding layer of this embodiment is implemented by a fully connected layer, and N in the formula represents the number of elements in the semantic confidence matrix.

[0062] Step S6, the bounding box results of the human and the object, the category of the object, and the verb classification result are used as the predicted triplet, the confidence score is used to constrain the predicted triplet, and the final detection result is obtained.

[0063] In the previous step S3, the bounding box results of the human and the object corresponding to the image to be detected and the category of the object have been obtained. In step S4, the verb classification result corresponding to the image to be detected is obtained. Thus, the predicted triplet can be obtained as follows:

[0064] Output<Human bbox , Object bbox , Object class , Verb class >

[0065] The output representing the prediction result is a triple, human bbox represents the bounding box of a person in the prediction result, Object bbox represents the bounding box of an object in the prediction result, Object class represents the class of the object in the prediction result, Verb class represents the class of the verb in the prediction result.

[0066] The embodiment adopts a confidence score to constrain the predicted triplets, which is specifically shown as follows:

[0067] If there is at least one predicted triplet with a confidence score greater than a confidence threshold, the predicted triplet with the confidence score greater than the confidence threshold is output as a detection result;

[0068] If the confidence scores corresponding to all predicted triplets are not greater than the confidence threshold, the predicted triplet with the highest confidence score is selected as the detection result.

[0069] In a specific embodiment, for a human-object relationship interaction detection network as shown in Figure 2 , the overall loss function thereof is composed of two parts, a first part is a detection loss, and a second part is a semantic loss:

[0070]

[0071] In the detection loss, the matching between the predicted target and the ground-truth is mainly completed through the Hungarian algorithm:

[0072]

[0073] wherein, The detection loss is composed of four parts, represents a bounding box regression loss, λ u represents an interaction joint Iou loss, represents a classification loss, represents a verb loss, λ b , λ u , λ c , λ a are learnable weight parameters.

[0074] Each specific loss is described as follows:

[0075] (1) The bounding box regression loss is specifically represented as follows:

[0076]

[0077] where Φ represents a total set of ground-truth, represents N q represents the total number of predicted bounding boxes, which is 100 by default in the present application, represents the prediction result of the human bounding box, represents the result of the ground-truth human bounding box, represents a function of mapping the index of the predicted value to the index of the ground-truth. represents the prediction result of the object bounding box. represents the difference between the predicted (human / object) bounding box and the ground-truth bounding box.

[0078] (2) The interactive joint IoU loss is specifically represented as follows:

[0079]

[0080] Most of the content represented is described in the bounding box regression loss, except that in the interactive joint IoU loss, the loss function body uses the GIoU function, and the IoU function is represented as IoU Intersection represents the intersection of the predicted box and the ground-truth box, Union represents the union of the predicted box and the ground-truth box, and the GIoU function is represented as A c is the maximum rectangle formed by the predicted box and the ground-truth box, u corresponds to the union area of the two boxes, and GIoU Loss = 1-GIoU, where 0≤GloU Loss≤2, so in in the interactive joint IoU loss that is, the loss calculation of the GIoU value of the predicted box and the ground-truth box.

[0081] (3) The classification loss is specifically represented as follows:

[0082]

[0083] where, represents the predicted class, which is represented by 0 and 1 to indicate whether it is the class, and Nobj This represents the number of object categories. This corresponds to the real object category value. This represents the loss value between the statistically predicted category and the actual category.

[0084] (4) The specific representation of verb loss is as follows:

[0085]

[0086] in, N act This indicates the number of types of verbs. This represents the number of predicted verb categories corresponding to the objects. f Focalloss, specifically represented as FL(p) t )=-α t (1-p t )γlog(p t ), α t To suppress the imbalance of positive and negative sample parameters, the γ parameter can control the imbalance of easy / hard sample numbers. t For the sample. Therefore, in In verb loss, l is used f To calculate the loss between the true verb category and the predicted verb category, where Indicates the category of real verbs.

[0087] (5) The semantic loss is specifically represented as follows:

[0088]

[0089] in, Among them, idx h This represents the category index corresponding to a person in the dataset, idx. obj idx represents the category index of the object in the dataset. act This represents the category index of the action in the dataset, and N represents the number of predicted triples. Represents a true and valid triplet. The sequence of triples predicted by the model is represented by a vector subtraction operation, and finally the sigmoid function is used to map it to 0 and 1 to complete the calculation of the loss value.

[0090] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for detecting semantic human-object interaction based on a cascaded decoder, the method comprising: constructing a human-object interaction detection network; and detecting semantic human-object interaction based on the human-object interaction detection network. The human-object interaction detection network comprises a backbone network, a shared encoder, a human-object decoder, a verb decoder, and a semantic branch, the semantic branch comprising a triple encoder, a sigmoid function, and a confidence encoding layer, the semantic human interaction detection method based on the cascade decoder comprising: The image to be detected is input into the backbone network to extract features, and the extracted features are flattened into one-dimensional vectors as image visual features for subsequent processing. The image visual features are input into the shared encoder to obtain human-object encoding features, shared encoding features, and visual encoding features. The human-object encoding features are input into the human-object decoder to obtain human-object decoding features, and then the human-object decoding features are input into a fully connected layer to obtain the bounding box results of the human and the object and the category of the object. The shared encoding features and the human-object decoding features are input into the verb decoder to obtain verb category features, and then the verb category features are input into a fully connected layer to obtain the verb classification results. The valid triple data of the image to be detected is input into the triple encoder to obtain triple encoding features, and then a semantic confidence matrix is constructed using the visual encoding features and the triple encoding features, and the semantic confidence matrix is subjected to a sigmoid function operation and then passed through a confidence encoding layer to obtain corresponding confidence scores. The bounding box results of the human and the object, the category of the object, and the verb classification results are used as predicted triples, the confidence scores are used to constrain the predicted triples, and finally the detection results are obtained. The valid triple data of the image to be detected comprises: The image to be detected is input into a pre-trained CLIP model to obtain valid triple data.

2. The method of claim 1, wherein, The shared encoder and the triple encoder both adopt a self-attention mechanism.

3. The method of claim 1, wherein, The human-object decoder and the verb decoder both adopt a deep self-attention transformer network.

4. The method of claim 1, wherein, The bounding box results of the human and the object, the category of the object, and the verb classification results are used as predicted triples, the confidence scores are used to constrain the predicted triples, and finally the detection results are obtained, comprising: If there is at least one predicted triple with a confidence score greater than a confidence threshold, the predicted triple with the confidence score greater than the confidence threshold is output as the detection result; If none of the confidence scores of all predicted triples is greater than the confidence threshold, the predicted triple with the highest confidence score is selected as the detection result.

5. The method of claim 1, wherein, The overall loss function of the human-object interaction detection network is composed of two parts, the first part is the detection loss, and the second part is the semantic loss: ； wherein the detection loss is represented as follows: ； wherein, The detection loss is composed of four parts, representing the bounding box regression loss, representing the interaction joint Iou loss, representing the classification loss, representing the verb loss, are weight parameters, respectively.

Images