A combined action recognition method based on a semantic decoupling self-attention model

Abstract

The application discloses a kind of based on the combination action recognition method of semantic decoupling self-attention model, and the most advanced effect is obtained on three kinds of division of STH-ELSE dataset.The method realizes the decoupling of the high-level semantic features of verb and object combination by object-verb decoupling module (OMD) and semantic decoupling constraint module (SDC), and alleviates the problem of model performance decline due to distribution deviation in combined action recognition.In OMD, several initialized learnable labels are set to capture the spatiotemporal features related to objects, and the learned spatiotemporal features are preliminarily decoupled in the high-level visual space.In SDC, text information is introduced to impose stricter consistency constraints on the decoupled features constructed in OMD at the semantic level, and finally the appearance and motion features after complete decoupling are learned.

Description

Technical Field

[0001] This invention relates to a combined action recognition method in the field of computer vision. Background Technology

[0002] Action recognition has become a research hotspot in computer vision due to its wide applications in human-computer interaction, virtual reality, and public safety. In recent years, with the continuous development of large-scale datasets and robust backbone networks, action recognition has experienced rapid advancements. However, traditional action recognition benchmark models perform poorly when encountering samples with unknown categories or distributions. This indicates that the models only learn strong appearance biases and cannot truly understand the semantic information of the actions themselves; their generalization ability is far from meeting the standards for real-world applications.

[0003] Human behavior is inherently combinable, and when the human visual perception system understands complex human actions, it typically doesn't focus on the objects themselves, but rather on the actions performed by the person on the object. Therefore, humans can easily infer unseen behavioral combinations from known combinations. To further challenge the generalization ability of models, the combined action recognition task has emerged. This task decomposes each action into a combination of a verb and one or more objects, and sets the verb-object combinations in the test set to not overlap with those in the training set, resulting in a different distribution of "verb-object" combinations between the training and test sets. This distribution shift aims to enhance the model's semantic understanding of the actions themselves, significantly challenging the model's generalization ability.

[0004] Overcoming the bias in the "action-object" combination across training and testing sets is a key challenge in combined action recognition. Previous studies have attempted to extract features from additional supervisory information, such as object bounding boxes and labels, to disrupt the original distribution of appearance features. However, these works cannot directly disrupt the combination of actions and objects; the features of actions and objects remain intertwined, hindering the model's understanding of the semantics of the actions themselves. Summary of the Invention

[0005] Purpose of the invention: To address the aforementioned existing technologies, this invention proposes a combined action recognition method based on a semantically decoupled self-attention model (DeFormer). This method solves the problem of performance degradation caused by distribution bias in combined action recognition, and enables the decomposition of verb and object combinations, thereby enhancing the model's ability to understand action semantics and its generalization capabilities.

[0006] Technical solution: A combined action recognition method based on a self-attention model with semantic decoupling, comprising the following steps:

[0007] Step 1: Randomly sample a certain number of frame images from the input video and obtain the object coordinates of each frame image; divide the frame images into blocks and embed them to obtain video tags; extract features from the frame images based on the object coordinates to obtain region tags; and use temporal aggregation coding to obtain object trajectory tags on the frame sequence.

[0008] Step 2: The tag obtained by adding the region tag and the trajectory tag is concatenated with the video tag. The concatenated tag is then input into the object-verb decoupling module to obtain the classification tag and the appearance and motion features after preliminary decoupling.

[0009] Step 3: Use a text encoder to encode the verbs and objects in the real video labels to obtain the corresponding text embeddings. Input the appearance and motion features obtained in Step 2 into the semantic decoupling constraint module, and calculate the comparison loss with the corresponding text embeddings after passing through the feature fusion unit.

[0010] Step 4: After fusing the features obtained in Step 2, calculate the classification loss with the real video labels, and add it to the appearance contrast loss and motion contrast loss obtained in Step 3 according to the weights to obtain the final model loss to optimize the network.

[0011] Step 5: Use the optimized network to recognize combined actions.

[0012] Furthermore, step 1 includes the following specific steps:

[0013] T frames with a resolution of H×W are randomly sampled from the input video as input. Obtain the O object coordinates of each frame image The sampled T frames are divided into blocks and embedded to obtain the video marker X. vid Block embedding is implemented using a single 3D convolution layer;

[0014] Feature extraction of the frame image based on object coordinates yields the region marker X. obj ,Right now:

[0015] X obj =MLP(MaxPool(RoIAlign(X) vid ,B)))+P obj

[0016] Where RoIAlign(·,·) is the region of interest alignment operation, MaxPool(·) is the max pooling layer, and MLP(·) is the multilayer perceptron, P obj Encode the object's position;

[0017] Obtaining object trajectory markers X on a frame sequence using temporal aggregation coding traj ,Right now:

[0018] X traj =ReLU(Conv1d(MLP(B)))+P traj

[0019] Where Conv1d is a 1D convolution, ReLU is the activation function, and P... traj Encode the trajectory position.

[0020] Furthermore, step 2 includes the following specific steps:

[0021] Mark the obtained region as X obj and trajectory marker X traj Added together with the video marker X vid Initialized classification tags The total video marker is obtained by splicing the images together. And input to the object-verb decoupling module; where C is the number of feature channels;

[0022] In the object-verb decoupling module, features are obtained after information interaction and feature extraction through multiple layers of self-attention. Right now:

[0023]

[0024] Among them, W q W k , For projection weights, Attn consists of several spatiotemporal attention layers, with the number of layers set to 8;

[0025] The obtained features In the first dimension, it is sequentially decomposed into classification labels after feature extraction. Video tag Regional trajectory mixed marking Will Assigning values ​​to classification features Will Each is assigned a value representing an appearance feature. and motion characteristics separately with Feature enhancement is performed at the cross-attention layer to obtain the enhanced appearance features. and enhanced motion characteristics Right now:

[0026]

[0027]

[0028] Among them, CA o For appearance features across attention layers, CA m For motion features, the attention layer is set to 1 for each layer.

[0029] Furthermore, step 3 includes the following specific steps:

[0030] A text encoder using frozen parameters encodes objects in video tags to obtain object text embeddings. Encode the verbs in the video tags to obtain verb text embeddings.

[0031] Enhanced appearance features and enhanced motion characteristics The input is fed into the semantic decoupling constraint module, where it undergoes feature fusion by the feature fusion unit to obtain the final appearance features. Final motion characteristics in For timing fusion unit, Both are spatial integration units, and each consists of two linear layers.

[0032] Using the obtained f tag f label For f o f m Perform semantic constraints. This represents the constraint relationship between semantic information and visual information; within a training batch, the corresponding semantic information in the same video... and Different videos are considered as positive sample pairs, and different videos are considered as negative sample pairs. The loss of a single video is the negative of the similarity between that video and the remaining videos in the same training batch, i.e.:

[0033]

[0034]

[0035] in, These are the final appearance features and object text embeddings of the i-th video in the same training batch, respectively. For the object text embedding in the j-th video within the same training batch, These are the final motion features and verb text embeddings for the k-th video in the same training batch, respectively. For the verb text embedding of the s-th video in the same training batch. Let be the appearance contrast loss for the i-th video in a training batch. The motion contrast loss is given by sim(·), where sim(·) is the similarity calculation function, and τ is the temperature parameter. [j≠i]∈{0,1} indicates that the function value is 1 when j≠i, O is the number of objects in a video, and N is the batch size.

[0036] Furthermore, step 4 includes the following specific steps:

[0037] Calculate the complete appearance contrast loss within the same training batch based on the loss of a single video. Compared with motion loss Right now:

[0038]

[0039]

[0040] Features and After fusion, the classification loss is calculated by comparing it with the true labels. Right now:

[0041]

[0042] Where CE is the cross-entropy loss function, y is the real label corresponding to the video; fusion(·) is the feature fusion function, which concatenates the three inputs in the first dimension;

[0043] The obtained appearance contrast loss motion contrast loss With classification loss The weights are summed to obtain the final model loss, which is used to optimize the network.

[0044]

[0045] Here, α, β, and γ are hyperparameters used to balance the three losses, and are set to 0.7, 0.2, and 0.1, respectively.

[0046] Beneficial Effects: This invention proposes a combined action recognition method based on a self-attention model with semantic decoupling, achieving state-of-the-art results on all three partitions of the STH-ELSE dataset. This method, through an Object-Verb Decoupling Module (OMD) and a Semantic Decoupling Constraint Module (SDC), decouples the high-level semantic features of verb and object combinations, mitigating the performance degradation caused by distribution bias in combined action recognition. In the OMD, several initial learnable tags are set to capture spatiotemporal features related to objects, and the learned spatiotemporal features are initially decoupled in a high-level visual space. Textual information is introduced into the SDC to impose stricter semantic-level consistency constraints on the decoupled features constructed in the OMD, ultimately learning the fully decoupled appearance and motion features. Attached Figure Description

[0047] Figure 1 Flowchart provided for this invention;

[0048] Figure 2 The model framework diagram provided for this invention;

[0049] Figure 3 , Figure 4 These are the experimental results of the present invention. Detailed Implementation

[0050] The invention will now be further explained with reference to the accompanying drawings.

[0051] To address the performance degradation caused by distribution bias in combined action recognition, this invention actively decouples object features and motion features in the high-level feature space and introduces text information for stricter semantic consistency constraints, thereby splitting verb and object combinations and enhancing the model's ability to understand action semantics and its generalization capabilities.

[0052] like Figure 1 , Figure 2 As shown, a combined action recognition method based on a self-attention model with semantic decoupling includes four processes: feature tag acquisition, verb-object decoupling, semantic decoupling constraints, and loss calculation.

[0053] Feature tag acquisition includes the following steps:

[0054] Step 1: Randomly sample T frames of H×W resolution from the input video as input. Obtain the O object coordinates of each frame image The sampled T frames are divided into blocks and embedded to obtain the video marker X. vid Block embedding is implemented using a single 3D convolution layer.

[0055] Step 2: Extract features from the frame image based on the object coordinates to obtain the region marker X. obj ,Right now:

[0056] X obj =MLP(MaxPool(RoIAlign(X) vid ,B)))+P obj

[0057] Where RoIAlign(·,·) is the region of interest alignment operation, MapPool(·) is the max pooling layer, and MLP(·) is the multilayer perceptron, P obj Encode the position of the object.

[0058] Step 3: Obtain the object trajectory marker X on the frame sequence using temporal aggregation coding. traj ,Right now:

[0059] X traj =ReLU(Conv1d(MLP(B)))+P traj

[0060] Where Conv1d is a 1D convolution, ReLU is the activation function, and P... traj Encode the trajectory position.

[0061] Verb-object decoupling includes the following steps:

[0062] Step 4: Mark the region obtained in Step 2 as X obj and the trajectory marker X obtained in step 3 traj Added together with the video marker X vid Initialized classification tags The total video marker is obtained by splicing the images together. And input it into the object-verb decoupling module; where C is the number of feature channels.

[0063] In the object-verb decoupling module, features are obtained after information interaction and feature extraction through multiple layers of self-attention. Right now:

[0064]

[0065] Among them, W q W k , For the projection weights, Attn consists of several spatiotemporal attention layers, with the number of layers set to 8. In the multi-layer self-attention network, each projection weight corresponds to a single linear layer, and the specific weight value is obtained after the network training is completed.

[0066] Step 5: Apply the features obtained in Step 4 In the first dimension, it is sequentially decomposed into classification labels after feature extraction. Video tag Regional trajectory mixed marking Will Assigning values ​​to classification features Will Each is assigned a value representing an appearance feature. and motion characteristics separately with Feature enhancement is performed at the cross-attention layer to obtain the enhanced appearance features. and enhanced motion characteristics Right now:

[0067]

[0068]

[0069] Among them, CA o For appearance features across attention layers, CA m For motion features, the attention layer is set to 1 for each layer.

[0070] Semantic decoupling constraints include the following steps:

[0071] Step 6: Use a text encoder with frozen parameters to encode objects in the video tags to obtain object text embeddings. Encode the verbs in the video tags to obtain verb text embeddings.

[0072] Step 7: Enhance the appearance features from Step 5 and enhanced motion characteristics The input is fed into the semantic decoupling constraint module, where it undergoes feature fusion by the feature fusion unit to obtain the final appearance features. Final motion characteristics in For timing fusion unit, Both are spatially integrated units, and each consists of two linear layers.

[0073] Step 8: Use f obtained in step 6 tag f label For f o f m Perform semantic constraints. This represents the constraint relationship between semantic information and visual information; within a training batch, the corresponding semantic information in the same video... and Different videos are considered as positive sample pairs, and different videos are considered as negative sample pairs. The loss of a single video is the negative of the similarity between that video and the remaining videos in the same training batch, i.e.:

[0074]

[0075]

[0076] in, These are the final appearance features and object text embeddings of the i-th video in the same training batch, respectively. For the object text embedding in the j-th video within the same training batch, These are the final motion features and verb text embeddings for the k-th video in the same training batch, respectively. For the verb text embedding of the s-th video in the same training batch. Let be the appearance contrast loss for the i-th video in a training batch. The motion contrast loss is given by sim(·), where sim(·) is the similarity calculation function, and τ is the temperature parameter. [j≠i] ∈{0,1} indicates that the function value is 1 when j≠i, O is the number of objects in a video, and N is the batch size.

[0077] Loss calculation includes the following steps:

[0078] Step 9: Calculate the complete appearance contrast loss within the same training batch based on the loss of the individual video in Step 8. Compared with motion loss Right now:

[0079]

[0080]

[0081] Step 10: Apply the features obtained in Step 5 and After fusion, the classification loss is calculated by comparing it with the true labels. Right now:

[0082]

[0083] Where CE is the cross-entropy loss function, y is the real label corresponding to the video, and fusion(·) is the feature fusion function, which concatenates the three inputs in the first dimension.

[0084] Step 11: Apply the appearance contrast loss obtained in Step 9 motion contrast loss The classification loss obtained in step 10 The weights are summed to obtain the final model loss, which is used to optimize the network.

[0085]

[0086] Here, α, β, and γ are hyperparameters used to balance the three losses, and are set to 0.7, 0.2, and 0.1, respectively.

[0087] Figure 3 This compares the results of our method with other methods on different STH-ELSE partitions, using TOP-1 accuracy as the evaluation metric. The values ​​in parentheses represent the percentage decrease in accuracy of the same method under different partitions.

[0088] Figure 4 The results of this method are compared with those of other methods when using real object coordinate frames, with TOP-1 and TOP-5 accuracy as evaluation metrics.

[0089] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for recognizing a combined action based on a semantic decoupling self-attention model, characterized in that, The method comprises the following steps: Step 1: randomly sampling T frame images from the input video, and obtaining object coordinates of each frame image; performing block embedding on the frame images to obtain video labels, performing feature extraction on the frame images according to the object coordinates to obtain region labels, and using time sequence aggregation coding to obtain object trajectory labels on the frame sequence; Step 2: splicing the labels obtained by adding the region labels and the trajectory labels together with the video labels, inputting the spliced labels into an object-verb decoupling module to obtain classification labels and appearance and motion features after preliminary decoupling; Step 3: using a text encoder to encode the verbs and objects in the video real labels respectively to obtain corresponding text embeddings, inputting the appearance and motion features obtained in step 2 into a semantic decoupling constraint module, and calculating a contrast loss after feature fusion units and corresponding text embeddings; Step 4: calculating a classification loss between the features obtained in step 2 after fusion and the video real labels, adding the appearance contrast loss and the motion contrast loss obtained in step 3 according to weights, and obtaining a final model loss to optimize the network; Step 5: using the optimized network to perform combined action recognition; The step 2 comprises the following specific steps: The obtained region is marked as X obj and the trajectory is marked as X traj After addition, the video is marked as X vid , the initialized classification mark The video total mark is obtained by splicing and input to the object-verb decoupling module; wherein C is the number of feature channels; In the object-verb decoupling module, the features after information interaction and feature extraction through multiple layers of self-attention That is: wherein W q , W k , is a projection weight, Attn is composed of several spatio-temporal attention layers, and the number of layers is set to 8; The obtained features The classification labels after feature extraction are sequentially disassembled in the first dimension Video labels Region track mixed labels Will be Assigned as classification features Will be Respectively assigned as appearance features And motion features Respectively again with Enter the cross-attention layer for feature enhancement, get enhanced appearance features And enhanced motion features That is: Wherein, CA o is the cross-attention layer of appearance features m is the cross-attention layer of motion features, and the number of layers is set to 1; The step 3 comprises the following specific steps: Encoding objects in a video tag using a text encoder with frozen parameters to obtain object text embeddings Encoding verbs in a video tag to obtain verb text embeddings enhanced appearance features and enhanced motion features are input into the semantic decoupling constraint module, and feature fusion is performed by the feature fusion unit respectively to obtain final appearance features final motion features wherein is a time sequence fusion unit, is a spatial fusion unit, both of which are composed of two linear layers; The obtained f tag , f label are subjected to semantic constraints, o , f m The semantic information represents the limiting relationship of the visual information; in one training batch, the same video corresponding to and are regarded as positive sample pairs, and different videos are regarded as negative sample pairs, and the loss of a single video is the opposite number of the similarity of the video and the remaining videos in the same training batch, that is:​ where, are the final appearance feature and object text embedding of the ith video in the same training batch, respectively, is the object text embedding of the jth video in the same training batch, are the final motion feature and verb text embedding of the kth video in the same training batch, respectively, is the verb text embedding of the sth video in the same training batch, is the appearance contrastive loss of the ith video in a training batch, is the motion contrastive loss of the kth video in a training batch, sim(·) is the similarity calculation function, τ is the temperature parameter, 1 [j≠i] ∈{0, 1} indicates that the function value is 1 when j≠i, O is the object number of a video, and N is the batch size.

2. The method of claim 1, wherein the method is a method of combined action recognition based on a self-attention model with semantic decoupling. The step 1 comprises the following specific steps: Randomly sample T frame images of resolution HxW from the input video as input Obtain O object coordinates of each frame image Block-embedding the sampled T frame images to obtain video label X vid The block-embedding is implemented by using one layer of 3D convolution. According to the object coordinates, the frame image is subjected to feature extraction to obtain region labels X obj That is: X obj = MLP(MaxPool(RoIAlign(X vid , B)) ) + P obj wherein RoIAlign(·,·) is a region of interest alignment operation, MaxPool(·) is a max-pooling layer, MLP(·) is a multi-layer perceptron, P obj is an object position encoding; Obtaining object trajectory labels X on a sequence of frames using temporal aggregation encoding traj i.e.: X traj = ReLU(Conv1d(MLP(B)) + P traj wherein Conv1d is a 1D convolution, ReLU is an activation function, P traj is the trajectory position encoding.

3. The method of claim 2, wherein the method further comprises: The step 4 comprises the following specific steps: The complete appearance contrast loss within the same training batch is calculated according to the loss of a single video and motion contrast loss That is: characteristics and fusion with the true label to calculate the classification loss that is: Wherein, CE is a cross-entropy loss function, y is a real label corresponding to the video; fusion(·) is a feature fusion function, that is, three inputs are dimensionally spliced in the first dimension; The appearance contrast loss obtained The motion contrast loss The classification loss The final model loss is obtained by adding the weights, that is: Wherein, α, β, γ are hyperparameters for balancing the three losses, and are respectively set to 0.7, 0.2 and 0.1.

Images