Character interaction detection method based on spatial context learning

Abstract

The invention provides a figure interaction detection method based on spatial context learning. The method comprises the following steps: S1, acquiring unprocessed visual features of an image as feature input of all subsequent decoders; s2, initializing instance query and spatial context query by using a spatial context exploration module; s3, respectively guiding a first decoder and a second decoder by utilizing instance query and spatial context query, and capturing instance features and spatial context features; and S4, according to the position codes, fusing the instance features and the spatial context features, initializing interaction action query, generating a character interaction detector, and predicting interaction features. According to the method, the problem that the performance is limited under the condition that the visual features of the instance are not obvious in the existing character interaction detection method is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence image processing, and in particular, to a method for detecting human interaction based on spatial context learning. Background Art

[0002] Human interaction detection aims to identify the interaction behaviors between instances (people and objects) in static images. Recently, transformer-based human interaction detectors have shown higher performance, but these methods mainly rely on the salient features of people and objects, such as appearance and pose.

[0003] This detector has the following problems: when instance features are unclear or occluded, these methods will significantly reduce the accuracy of interaction recognition, especially in difficult scenes where instance features are lost, occluded and blurred, and in complex scenes with multiple people and objects.

[0004] The difficulty of solving the above problems and defects is:

[0005] Most existing human interaction detection methods are implemented based on the object detection transformer framework. These methods usually inherit the inherent limitations of object detection transformers in processing complex scenes, which are difficult to overcome by enhancing features alone.

[0006] The significance of solving the above problems and defects:

[0007] The present invention provides a method for detecting human interaction based on spatial context learning, which can utilize learnable spatial context background area features rich in supporting semantics to assist the detector in making accurate predictions of human interaction in scenarios where instance features are damaged. It is foreseeable that the present invention can be used for monitoring and safety protection of various complex scene operations, and is robust to interference from severe weather and complex environments. Summary of the invention

[0008] The present invention provides a method for detecting human interaction based on spatial context learning, which solves the problem that the existing human interaction detection methods have limited performance when the visual features of instances are not significant.

[0009] The technical solution of the present invention is as follows:

[0010] The method for detecting human interaction based on spatial context learning of the present invention comprises the following steps: S1. obtaining unprocessed visual features of an image as feature inputs for all subsequent decoders; S2. initializing instance queries and spatial context queries using a spatial context exploration module; S3. guiding a first decoder and a second decoder respectively using instance queries and spatial context queries to capture instance features and spatial context features; and S4. initializing interactive action queries, generating a human interaction detector, and predicting interactive features based on fusion of instance features and spatial context features according to position coding.

[0011] Optionally, in the above-mentioned method for detecting human interaction based on spatial context learning, in step S1, a feature extractor is initialized using a convolutional neural backbone network pre-trained for target detection and a visual transformer encoder weight to obtain unprocessed visual features of the image. Among them, B, H ′ ,W ′ , C are the batch size, the dimension of the image after being processed by the feature extractor, and the latent space dimension of the feature extractor model.

[0012] Optionally, in the above-mentioned method for human interaction detection based on spatial context learning, in step S2, the text embeddings of instance categories and interaction categories are calculated using the offline text encoder of the visual-text pre-training model; the text embeddings are used to initialize the instance query branch and the context query branch of the context exploration module respectively, and these two branches use unprocessed visual features to predict instance feature responses and context feature responses respectively, and the responses are used to index and sort the unprocessed visual features for initializing instance queries and spatial context queries.

[0013] Optionally, in the above-mentioned person interaction detection method based on spatial context learning, the visual-text pre-training model is a visual-text pre-training model based on contrastive learning (CLIP)

[0014] Optionally, in the above-mentioned method for detecting human interaction based on spatial context learning, step S2 includes:

[0015] S2.1. When calculating text embedding, obtain the text annotation of the category corresponding to the training data set sample, construct the instance category text annotation and the interaction category text annotation; then, use the visual text pre-trained text encoder to predict the text embedding of the instance category text annotation and the interaction category text annotation, and the embedding is used to initialize the weights of the instance query branch and the context query branch of the spatial context exploration module respectively;

[0016] S2.2. According to the formula Calculate the confidence of visual-text matching and in, is the feature after the unprocessed visual feature is transformed into the CLIP text feature dimension by a single-layer linear layer. θinstance and θinstance are the linear layer weights initialized by instance text embedding and interactive text embedding, respectively. argmax represents the highest response to the prediction set Ωinstance and Ωcontext of the instance linear layer and the context linear layer along the category dimension, and obtains the instance feature response and the context feature response.

[0017] S2.3. According to the formula Using ω instances and ω contexts as indices, from H ′ ×W ′ Select the first N q The largest block fills instance queries and context queries Get instance query and context query.

[0018] Optionally, in the above-mentioned method for detecting human interaction based on spatial context learning, step S2 includes:

[0019] S3.1. Construct a first decoder, a second decoder and a third decoder, wherein the first decoder and the second decoder use the Q instance and the Q context obtained in step S2 as query inputs, respectively, and the first decoder and the second decoder predict instance features, first position representations, context features, and second position representations, respectively;

[0020] S3.2. Construct spatial contrast loss. The spatial contrast loss acts on instance features and context features. The spatial contrast loss includes feature-level, region-level and instance-level spatial contrast losses. According to the formula

[0021]

[0022] Construct a spatial contrast loss at the feature level, where the instance features predicted by the decoder and spatial context features The absolute cosine similarity along the query dimension is used to construct this loss, To query the value that is not empty on the dimension, according to the formula

[0023]

[0024] Construct a region-level spatial contrast loss where the instance position representation predicted by the decoder and contextual position representation The exponential of the inverse first-order distance is used to construct the loss, according to the formula

[0025]

[0026] Construct instance-level spatial contrast loss, where the regression prediction network is based on Predicted subject and object bounding boxes and regression prediction network based on The inverse generalized set intersection between predicted spatial context bounding boxes is used to construct the loss;

[0027] S3.3. Add the first position representation and the second position representation along the position encoding dimension as the query input of the third decoder, and the third decoder predicts the interaction feature.

[0028] Optionally, in the above-mentioned method for character interaction detection based on spatial context learning, in step S4, a group of multi-layer perceptron networks use instance features and interaction features to predict character interaction detection predictions, and the character interaction detection predictions include human body position, object position, object category and interaction category.

[0029] According to the technical solution of the present invention, the beneficial effects produced are:

[0030] The spatial context learning process of the present invention can effectively alleviate the detection problems in complex scenes, difficult scenes and pictures with people occluding images; in addition, the spatial context exploration module proposed in the present invention is transferable and can significantly improve the performance of most existing task interaction detectors based on transformers, while having no additional computational overhead in the reasoning process. Finally, the present method can achieve the best recent human interaction detection performance under reasonable training settings.

[0031] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0033] Figure 1 A schematic diagram of a process of a method for detecting human interaction based on spatial context learning according to the present invention;

[0034] Figure 2 A schematic diagram of the overall network structure of a preferred embodiment of a method for detecting human interaction based on spatial context learning according to the present invention;

[0035] Figure 3 for Figure 2 Schematic diagram of the workflow of the spatial context exploration module of the embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0037] like Figure 1 , Figure 2 As shown, the method for detecting human interaction based on spatial context learning of the present invention comprises the following steps:

[0038] S1. Obtain the unprocessed visual features of the image as the feature input for all subsequent decoders;

[0039] In step S1, the feature extractor is initialized using the convolutional neural backbone network pre-trained for object detection and the visual transformer encoder weights to obtain the unprocessed visual features of the image. Among them, B, H′, W′, and C are the batch size, the dimension of the image after being processed by the feature extractor, and the latent space dimension of the feature extractor model, respectively.

[0040] The unprocessed visual features of the image are obtained by: reading the input image into RGB representation format, using Figure 2 The feature extractor shown, such as but not limited to ResNet50+DETR encoder or Swin-Transformer+DN-DETR encoder, extracts visual feature representations to obtain unprocessed visual features The visual feature is Figure 2 The visual feature inputs of the first decoder, the second decoder and the third decoder.

[0041] S2. Initialize instance query and spatial context query using spatial context exploration module;

[0042] Specifically, the text embeddings of instance category text annotations and interaction category text annotations are calculated using the visual text pre-trained text encoder of the frozen visual-text pre-trained model; the text embeddings are used to initialize the instance query branch of the context exploration module and the context query branch of the context exploration module respectively. These two branches use unprocessed visual features to predict instance feature responses and context feature responses respectively, and the responses are used to index and sort unprocessed visual features for initializing instance queries and spatial context queries. The selected visual-text pre-trained model is the visual-text pre-trained model based on contrastive learning (CLIP). In order to obtain the best performance, CLIP ViT-L / 14 is used.

[0043] Step S2 specifically includes the following steps:

[0044] S2.1. When calculating text embedding, obtain the text annotations of the categories corresponding to the training dataset samples, such as Figure 3As shown, to construct instance category text annotations and interaction category text annotations, the original category labels or category descriptions that conform to the templates of "a picture of <object category>" and "a picture of a person <interacting category> an <object category>" can be used to reconstruct the instance category text annotations into "a picture of <instance category>" and the interaction category text annotations into "a picture of a person <interacting category> an <instance category>"; then, the visual text pre-trained text encoder is used to predict the text embeddings of the instance category text annotations and the interaction category text annotations, which are used to initialize the text embeddings as shown in FIG. Figure 3 The weights of the instance query branch and the context query branch of the spatial context exploration module.

[0045] S2.2. According to the formula

[0046]

[0047] Calculate the confidence of visual-text matching and in, is the feature after the unprocessed visual feature is transformed from a single linear layer to the CLIP text feature dimension, θ 实例 and θ 实例 are the linear layer weights initialized by instance text embedding and interaction text embedding respectively, and argmax represents the prediction set Ω of the instance linear layer and the context linear layer along the category dimension. 实例 and Ω 上下文 Take the highest response. Figure 3 The example feature responses and context feature responses are shown in .

[0048] S2.3. According to the formula Using ω 实例 and ω 上下文 As an index, select the first N from H′×W′ image blocks q The largest block fills instance queries and context queries From this we can get Figure 3 In the example query and context query shown in . In this embodiment, N q and C are set to 64 and 256.

[0049] S3. Use instance query and spatial context query to guide Figure 2 The first decoder and the second decoder in the proposed method capture instance features and spatial context features. The spatial contrast loss of the proposed method is used to constrain instance features and spatial context features, and back-propagation learning is performed.

[0050] This step includes:

[0051] S3.1. Construction Figure 2 The first decoder, the second decoder and the third decoder are shown. The decoder should select a transformer-based multi-layer decoder, such as a DETR decoder, a Swin-transformer decoder or a DN-DETR decoder. In this embodiment, the first decoder and the second decoder use a DETR decoder, and the third decoder uses a HOICLIP decoder combined with CLIP visual feature attention. The first decoder and the second decoder use the Q obtained in step S2 respectively. 实例 and Q 上下文 As query input, the first decoder and the second decoder predict instance features, first position representation and context features, second position representation respectively.

[0052] S3.2. Construct a spatial contrast loss, which acts on the instance features and context features obtained in step S3.1. The spatial contrast loss includes three types of losses: feature-level, region-level, and instance-level spatial contrast losses. According to the formula

[0053]

[0054] Construct a spatial contrast loss at the feature level, where the instance features predicted by the decoder and spatial context features The absolute cosine similarity along the query dimension is used to construct this loss, The value that is not empty in the query dimension. According to the formula

[0055]

[0056] Construct a region-level spatial contrast loss where the instance position representation predicted by the decoder and contextual position representation The exponential of the inverse first-order distance is used to construct the loss. According to the formula

[0057]

[0058] Construct instance-level spatial contrast loss, where the regression prediction network is based on Predicted subject and object bounding boxes and regression prediction network based on The Generalized Intersection of Union (GIoU) ​​between the predicted spatial context bounding boxes is used to construct the loss.

[0059] S3.3. Add the first position representation and the second position representation along the position encoding dimension as the query input of the third decoder, and the third decoder predicts the interaction feature.

[0060] S4. Based on the position encoding, the instance features and spatial context features are fused to initialize the interactive action query, generate the person interaction detector, and predict the interaction features. The above instance features and interaction features are used for person interaction detection and prediction.

[0061] A set of multi-layer perceptron networks predicts output predictions (including human position, object position, object category, interaction category) using instance features and interaction features. Similar to step S2.1, the multi-layer perceptron used in step S3.3 also uses instance and context text embedding for weight initialization in this embodiment.

[0062] Test experiment

[0063] The present invention has been tested on two classic large-scale datasets of human-object interaction detection tasks to verify the effectiveness of the invention. Test experiments of the fully supervised training model of the present invention were carried out on two tasks, HICO-DET and V-COCO, datasets of human-object interaction detection tasks, to verify performance. Specifically, performance tests were carried out on three partitions of the HICO-DET dataset, full partition, rare partition, non-rare partition and two experimental settings, default setting and known object setting. Performance tests were carried out on the scene 1 setting of the V-COCO dataset. Specifically, the mean average precision of the predicted results and the true annotations is calculated. This indicator measures the average accuracy of the prediction of interactive actions when the area intersection of the candidate box of the predicted instance and the bounding box of the true annotated instance is greater than a sufficient threshold. The verification results are shown in Table 1.

[0064] Table 1

[0065]

[0066]

[0067] It can be found from Table 1 that the human interaction detector obtained by the present invention has stronger detection performance and is robust to various scenarios.

[0068] The spatial context learning process of the present invention can effectively alleviate the detection problems in complex scenes, difficult scenes and pictures with people occluding images. In addition, the spatial context exploration module proposed in the present invention is transferable and can significantly improve the performance of most existing task interaction detectors based on transformers, while having no additional computational overhead in the reasoning process. Finally, the present method can achieve the best recent human interaction detection performance under reasonable training settings.

[0069] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for detecting human interaction based on spatial context learning, characterized in that: The following steps are involved: S1. Obtain the unprocessed visual features of the image as the feature input for all subsequent decoders; S2. Initialize instance query and spatial context query using spatial context exploration module; S3. Using the instance query and the spatial context query to guide the first decoder and the second decoder respectively, capturing instance features and spatial context features; as well as S4. The instance features and the spatial context features are fused according to the position encoding, an interactive action query is initialized, a character interaction detector is generated, and an interactive feature is predicted.

2. The method for detecting human interaction based on spatial context learning according to claim 1, characterized in that: In step S1, the feature extractor is initialized using the convolutional neural backbone network pre-trained for object detection and the visual transformer encoder weights to obtain the unprocessed visual features of the image. Among them, B, H ′ ,W ′ , C are the batch size, the dimension of the image after being processed by the feature extractor, and the latent space dimension of the feature extractor model.

3. The method for detecting human interaction based on spatial context learning according to claim 1, characterized in that: In step S2, the text embeddings of instance categories and interaction categories are calculated using the offline text encoder of the frozen visual-text pre-trained model; the instance query branch and the context query branch of the context exploration module are respectively initialized using the text embeddings, and the two branches use unprocessed visual features to predict instance feature responses and context feature responses respectively, and the unprocessed visual features are indexed and sorted using the responses for initializing instance queries and spatial context queries.

4. The method for detecting human interaction based on spatial context learning according to claim 3, characterized in that: The visual-text pre-training model is a contrastive learning based visual-text pre-training model (CLIP).

5. The method for detecting human interaction based on spatial context learning according to claim 3, characterized in that: Step S2 includes: S2.

1. When calculating text embedding, obtain the text annotation of the category corresponding to the training data set sample, construct the instance category text annotation and the interaction category text annotation; then, use the visual text pre-trained text encoder to predict the text embedding of the instance category text annotation and the interaction category text annotation, and the embedding is used to initialize the weights of the instance query branch and the context query branch of the spatial context exploration module respectively; S2.

2. According to the formula Calculate the visual-text matching confidence and in, is the feature after the unprocessed visual feature is transformed into the CLIP text feature dimension by a single-layer linear layer. θinstance and θinstance are the linear layer weights initialized by instance text embedding and interactive text embedding, respectively. argmax represents the highest response to the prediction set Ωinstance and Ωcontext of the instance linear layer and the context linear layer along the category dimension, and obtains the instance feature response and the context feature response. S2.

3. According to the formula Using ω instances and ω contexts as indices, from H ′ ×W ′ Select the first N q The largest block fills instance queries and context queries Get instance query and context query.

6. The method for detecting human interaction based on spatial context learning according to claim 1, characterized in that: Step S2 includes: S3.

1. Construct a first decoder, a second decoder and a third decoder, wherein the first decoder and the second decoder use the Q instance and the Q context obtained in step S2 as query inputs, respectively, and the first decoder and the second decoder predict instance features, first position representations, context features, and second position representations, respectively; S3.

2. Construct a spatial contrast loss, which acts on the instance feature and the context feature. The spatial contrast loss includes feature-level, region-level and instance-level spatial contrast losses. According to the formula Construct the spatial contrast loss of the feature level, where the instance features predicted by the decoder and spatial context features The absolute cosine similarity along the query dimension is used to construct this loss, To query the value that is not empty on the dimension, according to the formula Construct a region-level spatial contrast loss where the instance position representation predicted by the decoder and contextual position representation The exponential of the inverse first-order distance is used to construct the loss, according to the formula Construct instance-level spatial contrast loss, where the regression prediction network is based on Predicted subject and object bounding boxes and regression prediction network based on The inverse generalized set intersection between predicted spatial context bounding boxes is used to construct the loss; S3.

3. Add the first position representation and the second position representation along the position encoding dimension as the query input of the third decoder, and the third decoder predicts the interaction feature.

7. The method for detecting human interaction based on spatial context learning according to claim 1, characterized in that: In step S4, a group of multi-layer perceptron networks predicts a person interaction detection prediction using the instance features and the interaction features, where the person interaction detection prediction includes a body position, an object position, an object category, and an interaction category.

Images