Combined action recognition method and system based on multi-level feature interactive fusion

Abstract

This invention discloses a combined action recognition method and system based on multi-level feature interaction fusion. The method includes the following steps: position-to-appearance feature extraction, which uses position information to extract instance-centric joint features from low-level appearance information; semantic feature interaction, which further explores the semantic interaction between joint features and instance identity; semantic-to-position prediction, which maps semantic features back to a low-dimensional position space to achieve instance position prediction; and action category prediction, which aggregates instance-centric features to achieve action recognition. This invention can effectively fuse features from different sources and improve the accuracy of combined action recognition.

Description

Technical Field

[0001] The present invention relates to the field of action recognition technology in the field of computer vision, and in particular to a combined action recognition method and system based on multi-level feature interactive fusion. Background Art

[0002] Human action recognition aims to understand human movements from a given video sequence. In recent years, numerous deep learning methods have been applied to this field, with a preference for applying powerful backbone networks to extract video features for recognition. Due to the complexity of actions and the limited number of sampled frames, the features extracted by these methods often exhibit an inductive bias towards appearance, associating the action with a specific object or scene. This approach is effective for recognizing simple actions that rely on scene information, but is inadequate for actions with complex temporal structures.

[0003] When recognizing actions, humans focus not only on scene information but also on the temporal relationship between distances between objects in the scene. This ability enables humans to easily recognize unseen combinations of actions and objects. This concept of combined action recognition has been incorporated into model design. Obviously, this combined reasoning requires not only image features that represent the scene but also information from other inputs, such as object positions. Because features at different levels vary significantly in modality and dimensionality, effectively fusing these multiple levels of features is a key challenge in combined action recognition. Summary of the Invention

[0004] The purpose of the present invention is to provide a combined action recognition method and system based on multi-level feature interactive fusion, which can use an interactive method to fuse information from different sources and improve the accuracy of combined action recognition.

[0005] The technical solution to achieve the purpose of the present invention is: a combined action recognition method based on multi-level feature interactive fusion, which includes the following steps:

[0006] Step 1: Feature extraction from position to appearance, which uses position information to extract instance-centric joint features from low-level appearance information;

[0007] Step 2: Semantic feature interaction, further exploring the semantic interaction between joint features and instance identities;

[0008] Step 3: Semantic-to-position prediction: mapping the semantic features back to the low-dimensional position space to achieve instance position prediction;

[0009] Step 4: Action category prediction, aggregating instance-centric features to achieve action recognition.

[0010] A combined action recognition system based on multi-level feature interaction fusion includes a feature extraction module, a semantic feature intersection module, a semantic-to-position prediction module, and an action category prediction module, wherein:

[0011] The feature extraction module is used for position-to-appearance feature extraction, extracting instance-centered joint features from low-level appearance information based on instance position information;

[0012] The semantic feature interaction module is used for semantic feature interaction to obtain the semantic interaction between the joint feature and the instance identity;

[0013] The semantic-to-position prediction module is used for semantic-to-position prediction, mapping the semantic features back to the position space of the original dimension for instance position prediction;

[0014] The action category prediction module is used for action category prediction and aggregates instance-centered features for combined action recognition.

[0015] A combined action recognition system based on multi-level feature interaction fusion includes a feature extraction module, a semantic feature intersection module, a semantic-to-position prediction module, and an action category prediction module, wherein:

[0016] The feature extraction module is used for position-to-appearance feature extraction, extracting instance-centered joint features from low-level appearance information based on instance position information;

[0017] The semantic feature interaction module is used for semantic feature interaction to obtain the semantic interaction between the joint feature and the instance identity;

[0018] The semantic-to-position prediction module is used for semantic-to-position prediction, mapping the semantic features back to the position space of the original dimension for instance position prediction;

[0019] The action category prediction module is used for action category prediction and aggregates instance-centered features for combined action recognition.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] (1) A simple and unified combined action recognition system is proposed, in which different information sources (including appearance, position, and semantic features) can be interactively fused in multiple stages to promote the fusion of multimodal information;

[0022] (2) The present invention uses the auxiliary position prediction task to force the model to pay more attention to potential motion clues, thereby promoting the model's modeling of temporal information;

[0023] (3) The present invention explicitly models non-appearance features to improve the model's generalization ability for scenes and object appearances. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the method of the present invention.

[0025] Figure 2 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings:

[0027] Combine Figure 1 and Figure 2 ,A combined action recognition method based on multi-level feature interaction, including four processes of ,feature extraction from position to appearance, semantic feature interaction, ,prediction from semantic to position, and action category prediction.

[0028] The feature extraction from position to appearance includes the following steps:

[0029] Step 1) Sample T frames from the video sequence and use the backbone network I3D to extract the spatiotemporal appearance representation, that is, the dimension is The feature map, d fea is the number of channels;

[0030] Step 2) According to the coordinate trajectory, use RoIAlign to crop and scale the obtained feature map to obtain the appearance feature F of each instance app ;

[0031] Step 3) Convert instance coordinates and instance identities into high-dimensional vectors through mapping and embedding respectively, and connect them, and then obtain the non-appearance features F of the object through a multi-layer perceptron. non_app The instance identity here refers to the code C that determines whether the instance is a person or an object;

[0032] Step 4) Connect F app and F non_app The instance-centered joint representation F of the video is obtained.

[0033] Semantic feature interaction includes the following steps:

[0034] Step 5) Transmit spatial information between instance nodes. Given a time dimension t, information is transmitted between nodes in space through interaction functions. According to the different instance identities, three interaction functions are constructed, namely and Corresponding to person and person ε ss 、People and Things so 、Things and Things oo Three sets of instance pairs. The spatial interaction function is:

[0035]

[0036] in, and denote the basic joint features and spatial features of the i-th instance at time t, respectively, [·,·] denotes the connection operation, To represent the basic joint features, MLPs can be used;

[0037] Step 6) Transfer time information between instance nodes. Given the spatial feature G of the i-th instance i , further capturing the temporal dependency through RNN and fusing the instance features along the time dimension.

[0038]

[0039] Among them, Cat(·) is used to connect all the outputs from RNN along the last dimension. output; T (·) is an MLP used to encode the features of each concatenated instance; θ T Represents the parameters learned in RNN. For each video, the spatiotemporal features Z of N instances are Z = {Z1, Z2, ..., Z N} is used for the final action classification;

[0040] The semantic-to-position prediction process mainly includes the following steps:

[0041] Step 7) Predict the future state of each instance based on the observed features. Given the first t observed features of the i-th instance The t+1th state is predicted to be:

[0042]

[0043] Here we use T obs The number of frames observed by the predictor at each step, so RNN is used to model the temporal structure hidden in the data, and the final output is regarded as the predicted state The RNN here shares parameters with the RNN in step 6);

[0044] Step 8) Given the predicted state of the i-th instance at time t+1 Two linear layers are used to estimate its absolute position and relative position (the difference between the center of an instance in two consecutive frames). The formula is:

[0045]

[0046] in When t <T obs hour and Fill with real values.

[0047] Action category prediction mainly includes the following steps:

[0048] Step 9) Pool the spatiotemporal features Z of the N instances obtained in step 6) as video-level representations, and use softmax to obtain category probabilities. The category with the highest probability is the category to which the video belongs.

Claims

1. A combined action recognition method based on multi-level feature interactive fusion, characterized in that: The following steps are involved: Perform position-to-appearance feature extraction, extracting instance-centric joint features from low-level appearance information based on instance position information; Perform semantic feature interaction to obtain semantic interactions between joint features and instance identities; Perform semantic-to-position prediction and map the semantic features back to the original dimensional position space for instance position prediction; Perform action category prediction and aggregate instance-centric features for combined action recognition; The acquisition of semantic interaction between the joint feature and the instance identity specifically includes: To transfer spatial information between instance nodes, given the time dimension t, nodes in space transfer information between each other through interaction functions. According to the different instance identities, three interaction functions are constructed, namely and Corresponding to person and person ε ss 、People and Things so 、Things and Things oo For three sets of instance pairs, the spatial interaction function is: in, and denote the joint features and spatial features of the i-th instance at time t, respectively, [·,·] denotes the connection operation, Represents joint features; Perform time information transfer between instance nodes, given the spatial features of the i-th instance Further capture temporal dependencies through RNN and fuse the spatial features along the temporal dimension; Among them, Cat(·) is used to connect all the outputs from RNN along the last dimension. output; T (·) is an MLP used to encode the features of each concatenated instance; θ T Represents the learnable parameters in RNN. For each video, the spatiotemporal features Z of N instances are Z = {Z1, Z2, ..., Z N } is used for the final action classification.

2. The combined action recognition method based on multi-level feature interactive fusion according to claim 1 is characterized in that: The method of extracting instance-centric joint features from low-level appearance information based on instance location information specifically includes: Sample M frames from the video sequence and use a deep convolutional network to extract its spatiotemporal representation, resulting in a dimension of The feature map, H and W are the width and height of the sampled image respectively, d fea is the number of channels; According to the coordinate trajectory, the appearance feature F of each instance is obtained by cropping and scaling the obtained feature map through RoIAlign app ; The instance position coordinates and instance identity are transformed into high-dimensional vectors through mapping and embedding respectively and connected, and then the non-appearance features F of the object are obtained through a multi-layer perceptron. non_app ,Instance identity refers to the code C that determines whether the instance is a person or an object; Connection appearance feature F app and non-appearance features F non_app The instance-centered joint feature F is obtained.

3. The combined action recognition method based on multi-level feature interactive fusion according to claim 2 is characterized in that: The high dimension is 512 dimensions.

4. The combined action recognition method based on multi-level feature interactive fusion according to claim 1 is characterized in that: described Acquired using MLPs.

5. The combined action recognition method based on multi-level feature interactive fusion according to claim 1 is characterized in that: Mapping the semantic features back to the original dimensional position space to perform instance position prediction specifically includes the following steps: Given the first t observed features of the i-th instance The t+1th state is predicted to be: M obs represents the number of frames observed by the predictor at each step, RNN is used to model the temporal structure hidden in the data, and the final output is regarded as the predicted state Given the predicted state of the i-th instance at time t+1 Estimate its absolute position through two linear layers and relative position for: in For when t <M obs hour and Fill with real values.

6. The combined action recognition method based on multi-level feature interactive fusion according to claim 1 is characterized in that: The aggregation of instance-centric features for combined action recognition is specifically as follows: The spatiotemporal features Z of the N instances obtained by pooling are expressed at the video level, and the category probability is obtained by softmax. The category with the highest probability is the category to which the video belongs.

7. A combined action recognition system based on multi-level feature interactive fusion for implementing the method according to any one of claims 1 to 6, characterized in that: It includes feature extraction module, semantic feature intersection module, semantic to position prediction module and action category prediction module, among which: The feature extraction module is used for position-to-appearance feature extraction, extracting instance-centered joint features from low-level appearance information based on instance position information; The semantic feature interaction module is used for semantic feature interaction to obtain the semantic interaction between the joint feature and the instance identity; The semantic-to-position prediction module is used for semantic-to-position prediction, mapping the semantic features back to the position space of the original dimension for instance position prediction; The action category prediction module is used for action category prediction and aggregates instance-centered features for combined action recognition.

Images