Semi-supervised learning method for skeleton action recognition model

Abstract

The application discloses a skeleton action recognition model semi-supervised learning method, proposes a new pyramid self-attention aggregation learning (PSPL) framework, and learns more comprehensive action representation covering coarse-to-fine-grained information through contrast learning. In the PSPL, pyramid aggregation attention (PPA) is introduced, body-level attention maps, part-level attention maps and joint-level attention maps are aggregated from three levels of body parts and joints, and semantic information of the skeleton action is supplemented from coarse granularity to fine granularity; secondly, the similarity of body / part / joint level contrast features between joints and motion modalities is measured through coarse-to-fine-grained contrast loss (CCL).

Description

TECHNICAL FIELD

[0001] The present application relates to the field of action recognition in the field of computer vision, in particular to a semi-supervised learning method of an action recognition model. BACKGROUND

[0002] Action recognition in video understanding is a meaningful and challenging topic, which has attracted extensive attention due to its important role in human-computer interaction, intelligent video surveillance, video retrieval and other aspects. In recent years, a large number of studies have been conducted on skeleton-based action recognition. Some representative skeleton-based action recognition methods use convolutional neural networks (CNN) to learn spatial features by taking skeleton features as pseudo images, use recurrent neural networks (RNN) to learn temporal dependencies by taking skeleton features as time series features, and use graph convolution networks (GCN) to learn relationship features by taking the relationship between human joints as a graph topology structure. Recently, some transformer-based methods perform better in learning features by capturing the relationship between all joints through self-attention mechanisms. However, most of the above methods use a fully supervised training mode, which relies on a large amount of skeleton data annotation, usually requiring a large amount of manual and time cost. Therefore, various semi-supervised learning methods that learn features from a large amount of unlabeled skeleton data and a small amount of labeled skeleton data have received extensive attention, and a large number of studies have appeared in recent years. However, for the task of skeleton-based action recognition, most semi-supervised methods only learn joint-level skeleton action representations, ignoring some additional semantic information at the coarse-grained level. SUMMARY

[0003] The present application relates to the field of action recognition in the field of computer vision, in particular to a semi-supervised learning method of an action recognition model.

[0004] Technical scheme: A semi-supervised learning method of a skeleton action recognition model, comprising:

[0005] Step 1: Obtain a training data set of skeleton sequence actions, including labeled joint modality data and motion modality data, and unlabeled joint modality data and motion modality data;

[0006] Step 2: input the training data into the encoder module to obtain the corresponding joint features and motion features;

[0007] Step 3: input the joint features and motion features corresponding to the unlabeled data obtained in step 2 into the skeleton pyramid structure module respectively, and divide them into different granularities to obtain body-level features, part-level features and joint-level features corresponding to the joint modality and motion modality respectively;

[0008] Step 4: The tertiary features obtained in step 3 are respectively input into a pyramid aggregation attention module, first, the body level attention map, the part level attention map and the joint level attention map are respectively calculated by the self-attention mechanism, and then the joint modal and the motion modal are respectively corresponding to the body level aggregation feature, the part level aggregation feature and the joint level aggregation feature are obtained by aggregating the attention map from the body to the part to the joint.

[0009] Step 5: The tertiary aggregation features obtained in step 4 are respectively input into a coarse-to-fine-grained contrast loss module, first, the body level contrast feature, the part level contrast feature and the joint level contrast feature corresponding to the joint modal and the motion modal are obtained by pooling and multi-layer perception, and then the similarity of the contrast features of the corresponding levels between different modalities is measured by calculating the body level contrast loss, the part level contrast loss and the joint level contrast loss.

[0010] Step 6: The joint features and motion features corresponding to the labeled data obtained in step 2 are respectively input into a nonlinear recognition layer including average pooling, a fully connected layer and a softmax layer to obtain action representation, and model training is performed by recognition loss.

[0011] Step 7: The contrast loss of step 5 and the recognition loss of step 6 are integrated as a training target to optimize the model.

[0012] Beneficial effects: In the action recognition based on skeleton, the coarse-grained is more directly reflecting the semantic information than the fine-grained (such as the skeleton joint). Therefore, according to the natural structure of the human body, the human skeleton structure is systematically divided into a pyramid skeleton structure from coarse to fine composed of body level, part level and joint level; since the information of the body level, the part level and the joint level is not independent, the self-attention mechanism is used to aggregate the semantic information from the body to the part to the joint; finally, the multi-level features are compared by contrast learning covering all granularities to learn the potential information of the action. The specific points are as follows:

[0013] 1. The present application proposes a new pyramid self-attention aggregation learning (PSPL) framework, which simultaneously learns the body level, part level and joint level skeleton action representation of the joint modal and motion modal containing complementary semantic information through coarse-to-fine-grained contrast learning.

[0014] 2. The present application proposes a new pyramid aggregation attention (PPA) mechanism, which supplements the semantic information of the skeleton action from coarse-to-fine-grained (i.e. from body to part to joint) by aggregating the body level attention map, the part level attention map and the joint level attention map.

[0015] 3. The present application designs a new coarse-to-fine contrast loss (CCL), including body-level contrast loss, part-level contrast loss and joint-level contrast loss, to measure the similarity between body / part / joint-level contrast features of joint modalities and motion modalities from coarse to fine granularity. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the present application.

[0017] Figure 2 、 Figure 3 are experimental results of the present application. DETAILED DESCRIPTION

[0018] The present application will be further explained below in conjunction with the accompanying drawings.

[0019] As shown in Figure 1 , a semi-supervised learning method for a skeletal action recognition model, the skeletal action recognition model includes an encoder module and a nonlinear recognition layer including an average pooling (AP), a fully connected layer (FC) and a softmax layer, the semi-supervised learning method includes:

[0020] Step 1: obtaining a training data set of skeletal sequence actions, including labeled joint modality data and motion modality data, and unlabeled joint modality data and motion modality data.

[0021] Step 2: sending the labeled and unlabeled joint modality data x j and the motion modality data x m into the encoder for encoding operation to obtain corresponding joint features f j and motion features f m ; wherein, the input data x j , x m ∈R M×C×T×N , M represents the number of input data, C represents the channel number of each input data, T represents the frame number contained in each input data, and N represents the number of joint nodes contained in each frame of input data.

[0022] Step 3: in the present application, the N joint-level nodes on the body are clustered into P part-level nodes in space, the part-level nodes are for example hands, arms and feet, each part-level node includes multiple joint-level nodes; the P part-level nodes are further clustered into B body-level nodes in space, the body-level nodes are for example left upper limbs, left lower limbs and trunks, each body-level node contains two or more part-level nodes.

[0023] The joint features f j and the motion features f mSkeleton pyramid structure module is not input, different granularity is divided, and semantic information from coarse to fine is learned. Joint feature f j ∈R M×C×T×N is divided into body-level feature z j ∈R M×C×T×B , part-level feature h j ∈R M×C×T×P and joint-level feature g j ∈R M×C×T×N ; motion feature f m ∈R M×C×T×N is divided into body-level feature z m ∈R M×C×T×B , part-level feature h m ∈R M×C×T×P and joint-level feature g m ∈R M×C×T×N .

[0024] Step 4: input the three-level features obtained in step 3 into the pyramid aggregation attention module, first calculate the body-level attention map, the part-level attention map and the joint-level attention map by the self-attention mechanism respectively, and then obtain the body-level aggregation feature, the part-level aggregation feature and the joint-level aggregation feature corresponding to the joint modal and the motion modal by aggregating the attention maps from the body level to the part level to the joint level. Specifically:

[0025] 1) the body-level feature z j / m is mapped to query feature Q, key feature K and value feature V through a linear projection layer:

[0026]

[0027] wherein W Q , W K , W V ∈R TC×TC are weight parameter matrices, and Q, K, C=S×C e , S is a parameter of the number of multiple heads in attention, and C e is the channel number of each head in the multiple attention.

[0028] 2) the body-level attention map A z is used to assign weights to each node at the body level, and important body-level information is emphasized in a self-attention manner, wherein A z ∈R M×S×B×B , the superscript H represents the rank of transformation; tanh(·) is used instead of softmax(·) to make the generated attention mapping not be constrained to be positive. The part-level attention map A hJoint-level attention diagram A g .

[0029] 3) Use the body-level attention map A obtained in 2) z To calculate body-level aggregated features Where σ is the activation function, ψ is a feedforward network consisting of linear layers containing BN functions, and Concat is used to connect the multi-head attention results.

[0030] 4) Aggregate semantic information from the body level to the part level:

[0031] First, let A z Fill with A z->h :

[0032] A z ={ω m,n 1≤m≤B 1≤n≤B

[0033] vi ,j =ω Ф(i),Ф(j) 1≤Ф(i)≤B 1≤Ф(j)≤B

[0034] A z->h ={v i,j} 1≤i≤P 1≤j≤P

[0035] Where, ω m,n ω m,n Represents body-level attention map A z The parameter with coordinates (m,n) represents the membership relationship between part-level nodes and body-level nodes, Ф(·). i,j To represent the attention map A after dimensional transformation z->h The parameters are in the coordinates (i,j).

[0036] Then combine A z->h and A h To obtain location-level aggregated features

[0037] A h,z =A h +λA z->h

[0038]

[0039] Where λ is the control A z->h The hyperparameter of the weight, A h,z For body-level attention maps, then part-level attention maps are generated.

[0040] Intentional diagram, A h,z ∈R M×S×P×PThe information of the part level and the body level is combined, and the key information of the action is comprehensively focused.

[0041] 5) The semantic information from the part level to the joint level and from the body level to the joint level is aggregated:

[0042] Firstly, according to the membership relationship of the joint level node and the part level node or the body level node, the A h->g and A z->g are filled in according to the method of step 4); then, A g , A h->g , A z->g are combined to obtain the joint level aggregated feature A

[0043] A g,h,z =A g +αA h->g +βA z->g

[0044]

[0045] wherein, α is a hyperparameter for controlling the weight of A h->g , β is a hyperparameter for controlling the weight of A z->g , and A g,h,z is the joint level attention graph after aggregating the body level attention graph and the part level attention graph, A g,h,z ∈R M×S×N×N The information of the joint level, the part level and the body level is aggregated, and the key information of the action is comprehensively focused.

[0046] Step 5: In order to facilitate the encoder to learn the joint modal and the complementary semantic information of the body level, the part level and the joint level in the joint modal from a large amount of unlabeled data, a new coarse-to-fine contrast loss (CCL) is adopted.

[0047] 1) The three-level aggregated features obtained in step 4 are respectively input into the coarse-to-fine contrast loss module, and the body level contrast features, the part level contrast features and the joint level contrast features corresponding to the joint modal and the motion modal are obtained through pooling and multi-layer perceptron (MLP) first. Specifically, the body level aggregated feature A The part level aggregated feature A and the joint level aggregated feature A are transformed into the body level contrast feature A The part level contrast feature A and the joint level contrast feature A

[0048] 2) Measure the similarity of the corresponding level contrastive features between different modalities by calculating the body level contrastive loss, the part level contrastive loss and the joint level contrastive loss.

[0049] Specifically, the body level contrastive loss of and is calculated as

[0050]

[0051] where μ i and represent the contrastive features from the same data but different modalities, i.e. μ i comes from the joint modality, comes from the motion modality, represents whether k and i are equal, 1 when k and i are not equal, and 0 when they are equal, when the batch number k in the summation formula equals the batch i in the outer summation formula, i.e. when μ i = μ k , the summation formula in the denominator skips the calculation of exp(sim(μ i , μ k ) / τ), where τ is a hyperparameter. The part level contrastive loss of and is calculated in the same way. The joint level contrastive loss of and is calculated in the same way.

[0052] Step 6: For the learning task of the skeletal action recognition model in the semi-supervised scenario, the entire model is jointly trained using the contrastive loss of unlabelled data and the recognition loss of labelled data. For a small amount of labelled data, the joint features and motion features corresponding to the labelled data obtained in step 2 are respectively input into the nonlinear recognition layer to obtain the action representation, i.e. the classification prediction label is obtained. The model is trained through the recognition loss .

[0053]

[0054]

[0055] where y represents the true label, and the recognition loss is, for example, the cross-entropy loss.

[0056] Step 7: The contrastive loss of step 5 and the recognition loss of step 6 are integrated, and the final training target of the model is where Optimizing the model as a training target.

[0057] As shown in Figure 2 and Figure 3 , the proposed method of the present application is compared with other existing typical methods in the skeleton-based semi-supervised recognition task. Compared with most other methods, the proposed PSPL is verified to perform well in the semi-supervised recognition task, proving the effectiveness of PSPL.

[0058] The comparison results of action recognition between various methods on the NTURGB+D dataset are shown in Figure 2 . Compared with semi-supervised learning-based methods, on the 10% CS benchmark, PSPL (accuracy of 78.1%) is 13.8% higher than the state-of-the-art ASSL (accuracy of 64.3%). Compared with unsupervised methods, on the 20% CS benchmark, PSPL (accuracy of 82.6%) is 3.9% higher than the state-of-the-art CMD (accuracy of 78.7%).

[0059] The comparison results of action recognition between various methods on the NW-UCLA dataset are shown in Figure 3 . Compared with semi-supervised methods, PSPL can be comparable to the state-of-the-art X-CAR and superior to other alternative methods. Compared with unsupervised methods, on the 15% semi-supervised setting, PSPL (accuracy of 77.8%) is 17.3% higher than the state-of-the-art MS 2 L (accuracy of 60.5%).

[0060] The above only describes the preferred embodiments of the present application, and it should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A semi-supervised learning method of a skeletal action recognition model, characterized in that, Comprise: Step 1: Obtain the training data set of skeleton sequence action, including labeled joint modal data and motion modal data, and unlabeled joint modal data and motion modal data; Step 2: Input the training data into the encoder module to obtain the corresponding joint features and motion features ; Step 3: Input the joint features and motion features corresponding to the unlabeled data obtained in step 2 into the skeleton pyramid structure module respectively, and divide them into different granularities to obtain the body level features, part level features and joint level features corresponding to the joint modal and motion modal respectively; Step 4: The tertiary features obtained in step 3 are respectively input into a pyramid aggregation attention module, first, the body-level attention map, the part-level attention map and the joint-level attention map are respectively calculated by the self-attention mechanism, and then the body-level aggregated features, the part-level aggregated features and the joint-level aggregated features corresponding to the joint modality and the motion modality are obtained by aggregating the attention maps from the body level to the part level to the joint level. ​​​ Step 5: Input the three-level aggregated features obtained in step 4 into the coarse-grained to fine-grained contrast loss module, first pass through pooling and multilayer perception to obtain the body level contrast features, part level contrast features and joint level contrast features corresponding to the joint modal and motion modal respectively, and then calculate the body level contrast loss, part level contrast loss and joint level contrast loss to measure the similarity of the corresponding level contrast features between different modalities; Step 6: Input the joint features and motion features corresponding to the labeled data obtained in step 2 into a nonlinear recognition layer including average pooling, one fully connected layer and softmax layer respectively to obtain the action representation, and train the model through recognition loss; Step 7: Integrate the contrast loss of step 5 and the recognition loss of step 6 as the training target to optimize the model; In step 3, the N joint level nodes on the body are clustered into P part level nodes in space, and the P part level nodes are further clustered into B body level nodes in space; Joint features are divided into body-level features , part-level features and joint-level features , motion features are divided into body-level features , part-level features and joint-level features ; In step 4, the body-level features are first mapped The query feature Q, the key feature K and the value feature V are obtained by linear projection layer mapping, and the body-level attention map , wherein the superscript H represents the rank of transformation, T is the number of frames, is the number of channels of each head in the multi-head attention; the same method is used to obtain the part-level attention map and the joint-level attention map ; In step 4, the body-level attention map is used to compute the body-level aggregated features wherein is an activation function, is a feed-forward network consisting of linear layers with BN functions.​ Aggregating semantic information from body level to part level: will be filled in as fill in as : wherein, a body-level attention map parameters with coordinates (m, n) in the map, a membership relation between a part-level node and a body-level node, a converted attention map parameters with coordinates (i, j) in the map; Then combine and to get the part-level aggregated features : wherein, is a control weight of the super parameter, is a part-level attention map after aggregating the body-level attention map Aggregating semantic information from part level to joint level and from body level to joint level: Firstly, the joint-level nodes are filled according to the membership relationship with the part-level nodes or the body-level nodes to obtain and ; then, combined with , , the joint-level aggregated features are obtained. wherein, is a control weighting hyperparameter, is a control weighting hyperparameter, is a joint-level attention map after aggregating the body-level attention map and the part-level attention map.

2. The method of claim 1, wherein, In step 2, the labeled and unlabeled joint modality data and motion modality data are respectively sent into encoders for encoding operations to obtain corresponding joint features and motion features .

3. The method of claim 1, wherein, The body-level aggregated features in step 5 are transformed into body-level contrastive features , part-level aggregated features and joint-level aggregated features in the contrast space , part-level contrastive features and joint-level contrastive features ; Computing and the body-level contrastive loss : Where M is the batch size; and These represent contrastive features from the same data but different modalities, i.e. Derived from joint modes, From motion modes, ; ; This indicates whether k and i are equal. A value of 1 indicates that k and i are not equal, and a value of 0 indicates that k and i are equal. The value is equal to the batch number k in the denominator summation formula compared to the batch number i in the external summation formula. = When the summation formula in the denominator is skipped The calculation, ; For hyperparameters; calculate using the same method. and Partial level comparison loss ,as well as and Joint-level contrast loss .

4. The method of claim 3, wherein, The final training objective of the model in step 7 ; wherein, , is the recognition loss value outputted by step 6.

Images