A local feature auxiliary based self-supervised action recognition method and system

Abstract

The application relates to the technical field of action recognition, and discloses a self-supervised action recognition method and system based on local feature assistance, which comprises the following specific steps: obtaining an unlabeled skeleton sequence; performing data enhancement processing to obtain an enhanced view; inputting into a graph convolution recognition network; obtaining global features in a global encoder and inputting the global features into a predictor to obtain prediction features; constructing a global loss; dividing the enhanced view into upper limb joints and lower limb joints and inputting into corresponding local encoders to obtain local features; constructing a local loss; obtaining saliency features and non-saliency features based on the global features through a self-attention mechanism; constructing saliency and non-saliency loss; training the graph convolution recognition network; and performing action recognition through the trained graph convolution recognition network. The application solves the problem that local information is not fully utilized in the prior art, and has the characteristics of effectively capturing distinguishing representations and strong learning ability.

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, and more specifically, to a self-supervised action recognition method and system based on local feature assistance. Background Technology

[0002] Early action recognition methods primarily modeled the spatiotemporal features of videos using hand-designed features, including both local and global features. With the development of deep learning, deep learning-based action recognition methods have gained widespread attention and achieved remarkable results. Due to the ubiquitous availability of RGB data, RGB-based action recognition became the mainstream research direction for a long time. Compared to RGB data, skeleton data represents the topological structure of the human body using joints and bones as units, exhibiting strong robustness to complex backgrounds, lighting, and viewpoint changes, and possessing advantages such as low computational complexity and low memory consumption.

[0003] Traditional deep learning-based action recognition methods possess powerful representational capabilities, but they all rely on large amounts of labeled data. Labeling and filtering each piece of data requires manual work, incurring significant costs to create a reasonably large-scale dataset. This manual labeling method cannot keep pace with the rapid generation of data on the internet. Self-supervised learning, on the other hand, can extract latent supervised information from large-scale unlabeled data and train the network using this constructed supervised information, thereby learning representations valuable for downstream tasks.

[0004] Mainstream self-supervised learning methods include generative and contrastive approaches. Generative methods primarily focus on reconstruction error, while contrastive learning methods do not require the model to reconstruct the input. Instead, the model maps similar inputs to neighboring regions in the feature space and dissimilar inputs to non-neighboring regions, thus achieving input classification. Compared to the reconstruction error of generative methods, contrastive methods are better able to uncover the consistency of data in classification tasks.

[0005] Existing technology offers a first-person perspective interactive action recognition method based on the fusion of global and local networks. This method includes: sampling videos to obtain different actions, resulting in image-based action samples; standardizing the size of the sampled action segments and performing data augmentation; training a 3D convolutional network based on global images as input to learn the spatiotemporal features of global actions and obtain a network classification model; using sparse optical flow to locate local regions where salient actions occur within the action segments; standardizing the size of the local regions of different actions, adjusting the network's hyperparameters, training a 3D convolutional network based on local images as input, and learning local salient action features to obtain a network classification model; sampling the same video multiple times to obtain action samples, and statistically ranking the prediction frequency given by the global and local models using a voting method, with the category with the most predictions being used as the recognized action label.

[0006] Most current methods for skeleton motion recognition only focus on the overall information of the skeleton, neglecting its local information. The dynamic changes of specific body parts contained in the local information play a decisive role in the classification of some actions; therefore, the correlation between local features should be considered in motion recognition and classification tasks.

[0007] Current self-supervised action recognition algorithms follow the MoCoV1 framework, employing multiple Siamese networks to model different skeleton modalities. This requires a large-scale negative sample memory to store negative samples for comparative learning with positive samples. Furthermore, most skeleton action recognition methods focus primarily on the overall information of the skeleton, neglecting local information. This local information includes dynamic changes in specific body parts, which play a crucial role in classifying certain actions.

[0008] In summary, existing technologies have the problem of not fully utilizing local information. Therefore, how to invent a self-supervised action recognition method based on local features is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] To address the problem that existing technologies do not fully utilize local information, this invention provides a self-supervised action recognition method based on local feature assistance, which has the characteristics of effectively capturing discriminative representations and having strong learning capabilities.

[0010] To achieve the above-mentioned objectives of this invention, the technical solution adopted is as follows:

[0011] A self-supervised action recognition method based on local feature assistance includes the following specific steps:

[0012] Obtain the unlabeled skeleton sequence;

[0013] Data augmentation is performed on the unlabeled skeleton sequence to obtain an augmented view;

[0014] The enhanced view is input into the graph convolutional recognition network;

[0015] Global features are obtained by sharing the same weights in a global encoder, and these global features are then input into a predictor to obtain predicted features. A global loss is constructed based on the global features and the predicted features.

[0016] The enhanced view is divided into upper limb joints and lower limb joints. The upper limb joints and lower limb joints are input into the corresponding local encoders to obtain local features. A local loss is constructed based on the local features.

[0017] By employing a self-attention mechanism, salient and non-salient features are obtained based on global features; explicit and implicit losses are then constructed based on these salient and non-salient features.

[0018] A graph convolutional recognition network is trained based on global loss, local loss, and explicit and implicit losses.

[0019] Action recognition is performed using a pre-trained graph convolutional recognition network.

[0020] Preferably, the graph convolutional recognition network adopts the spatiotemporally adaptive graph convolutional network STAGCN; specifically, STAGCN is represented as:

[0021]

[0022] Where z represents the feature, A k B is an unlearnable adjacency matrix; k and C k It is A k The initialized learnable matrices, α and β, are used to balance B. k and C k Hyperparameters.

[0023] Furthermore, the graph convolutional recognition network updates parameter B through backpropagation during training. k and C k .

[0024] Furthermore, the data augmentation processing of the unlabeled skeleton sequence to obtain augmented views specifically involves: processing the unlabeled skeleton sequence using data augmentation methods including shearing, cropping, flipping, and coordinate axis occlusion to obtain different augmented views {x}. i |i=1,2}.

[0025] Furthermore, specifically, the global loss, local loss, explicit and implicit losses all employ the InfoNCE loss function; the InfoNCE loss function is expressed as:

[0026]

[0027] Here, x and y are positive sample pairs from the same sample, and x and z are negative sample pairs from different samples; B(y) represents the set of all samples containing y in the current sample batch, and τ is a temperature hyperparameter used to balance the training process.

[0028] Furthermore, the specific steps for constructing the global loss are as follows:

[0029] Enhance view {x i The inputs of |i=1,2} to a global encoder sharing the same weights yield global features {z}. i |i=1,2};

[0030] global feature {z i |i=1,2} are processed by a predictor consisting of two fully connected layers to generate corresponding predicted features p1 and p2;

[0031] Consider p1 as a positive sample pair of z2, p2 as a positive sample pair of z1, and the other samples as negative sample pairs.

[0032] A prediction task replaces the task of comparing global features and predicted features; by mapping global features to the prediction feature space, the prediction relationship between corresponding actions is established to maximize the consistency between global features and predicted features; a global loss is constructed based on global features and predicted features.

[0033] L global =L Info (z1,p2)+L Info (z2,p1).

[0034] Furthermore, a local loss is constructed, with the following specific steps:

[0035] {x i |i=1,2} are divided into upper limb joints and lower limb joints

[0036] Each and The input is fed into the corresponding local encoder to obtain the local features of the upper limb. and local features of the lower limbs

[0037] Constructing local loss:

[0038]

[0039]

[0040] in, and These are positive sample pairs, {i,j}={1,2}.

[0041] Furthermore, through a self-attention mechanism, salient and non-salient features are obtained based on global features; explicit and implicit losses are then constructed based on these salient and non-salient features. The specific steps are as follows:

[0042] Employing the self-attention mechanism of Transformer, utilizing weight W Q W K W V z i Mapped to {Q,K,V}:

[0043] Q = z i W Q K = z i W K V=z i W V

[0044] The similarity between features is calculated using the Q and K matrices, then mapped to a low-dimensional space using V, and finally a salient region attention mask M is obtained through a linear layer and a sigmoid function. s Non-salient region attention mask M ns :

[0045]

[0046] M ns =IM s

[0047] Where, d k λ represents the feature dimension and is used for normalization; λ is a hyperparameter used to adjust the degree of fluctuation in feature changes during training.

[0048] Through M s and M ns Calculate the significance feature q s Non-significant features q ns :

[0049]

[0050]

[0051] The salient features q are processed through a multilayer perceptron (MLP) layer. s Non-significant features q ns Mapped to a lower-dimensional space;

[0052] The salient features of the two views are used as positive sample pairs, and the global feature z is used as the positive sample pair. iThe non-significant features are used as latent negative sample pairs to construct explicit and implicit losses:

[0053]

[0054]

[0055] in and Each is a positive sample pair, {i,j}={1,2}.

[0056] Furthermore, when training a graph convolutional recognition network, the loss function is based on global loss, local loss, and explicit and implicit losses:

[0057] L total =L global +L up +L low +L s +L ns .

[0058] A self-supervised action recognition system based on local feature assistance includes a cascaded data acquisition module, a data augmentation module, a graph convolutional network module, a training module, and an action recognition module.

[0059] The data acquisition module is used to acquire unlabeled skeleton sequences;

[0060] The data augmentation module performs data augmentation processing on the unlabeled skeleton sequence to obtain an enhanced view;

[0061] The graph convolutional network module is used to obtain global features through a global encoder sharing the same weights, and input the global features into a predictor to obtain predicted features; a global loss is constructed based on the global features and predicted features; the augmented view is divided into upper limb joints and lower limb joints, and the upper limb joints and lower limb joints are input into the corresponding local encoders to obtain local features; a local loss is constructed based on the local features; salient and non-salient features are obtained based on the global features through a self-attention mechanism; explicit and implicit losses are constructed based on the salient and non-salient features.

[0062] The action recognition module is used to perform action recognition through a trained graph convolutional recognition network.

[0063] The beneficial effects of this invention are as follows:

[0064] This paper discloses a self-supervised action recognition method based on local feature assistance. Utilizing a graph convolutional recognition network, it considers global features, local features, and explicit and implicit losses. Thus, it leverages local information from the overall skeleton and explicit local features from global features. Compared to traditional self-supervised action recognition methods that require a large number of negative samples, this invention optimizes the memory consumption of the negative sample memory bank, solves the problem of insufficient utilization of local information in existing technologies, and features effective capture of discriminative representations and strong learning ability. Attached Figure Description

[0065] Figure 1 This is a schematic diagram of the flowchart of a self-supervised action recognition method based on local feature assistance according to the present invention.

[0066] Figure 2 This is a schematic diagram of the algorithm framework of the self-supervised action recognition method based on local feature assistance.

[0067] Figure 3 This is a global feature framework diagram of a self-supervised action recognition method based on local feature assistance.

[0068] Figure 4 This is a local feature framework diagram of the self-supervised action recognition method based on local feature assistance. Detailed Implementation

[0069] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0070] Example 1

[0071] like Figure 1 As shown, a self-supervised action recognition method based on local feature assistance includes the following specific steps:

[0072] Obtain the unlabeled skeleton sequence;

[0073] Data augmentation is performed on the unlabeled skeleton sequence to obtain an augmented view;

[0074] The enhanced view is input into the graph convolutional recognition network;

[0075] Global features are obtained by sharing the same weights in a global encoder, and these global features are then input into a predictor to obtain predicted features. A global loss is constructed based on the global features and the predicted features.

[0076] The enhanced view is divided into upper limb joints and lower limb joints. The upper limb joints and lower limb joints are input into the corresponding local encoders to obtain local features. A local loss is constructed based on the local features.

[0077] By employing a self-attention mechanism, salient and non-salient features are obtained based on global features; explicit and implicit losses are then constructed based on these salient and non-salient features.

[0078] A graph convolutional recognition network is trained based on global loss, local loss, and explicit and implicit losses.

[0079] Action recognition is performed using a pre-trained graph convolutional recognition network.

[0080] Example 2

[0081] In one specific embodiment, such as Figure 2 As shown, to address the shortcomings of existing self-supervised action recognition algorithms that lack the use of local information, this invention proposes an adaptive self-supervised action recognition framework, AdaSCLR, which extracts key action information using both global and local features. Here, the Global Encoder is the global encoder, the Predictor is the predictor, the Local Encoder is the local encoder, and the Attention Mask is the self-attention mask. First, data augmentation methods such as shearing, cropping, flipping, and coordinate axis occlusion are used to process the unlabeled skeleton sequence, obtaining different augmented views {x}. i |i=1,2}. Then {x i The inputs of |i=1,2} to a global encoder sharing the same weights yield global features {z}. i |i=1,2}, and input the global features into the predictor to obtain the predicted features {p i |i=1,2}. To obtain locally explicit features of global information, the method in this chapter utilizes a self-attention mechanism to process global features, thereby obtaining salient features. Non-significant features

[0082] In addition to the global information mentioned above, this invention also models local information to help enhance the network's representation learning ability. First, {x} i |i=1,2} are divided into upper limb joints and lower limb joints Next, respectively and The data is input into a local encoder. Since the number of nodes differs between the upper limb and lower limb, different local encoders are used to encode the information of the upper limb and lower limb respectively, ultimately yielding the local features of the upper limb. and local features of the lower limbs

[0083] This invention uses the InfoNCE loss function as the loss function for the entire framework to calculate feature consistency:

[0084]

[0085] In the formula, x and y are positive sample pairs from the same sample, while x and z are negative sample pairs from different samples. B(y) represents the set of all samples containing y in the current sample batch, and τ is a temperature hyperparameter used to balance the training process.

[0086] Example 3

[0087] More specifically, the graph convolutional recognition network mentioned above adopts the spatiotemporally adaptive graph convolutional network STAGCN, which is specifically represented as:

[0088]

[0089] Where z represents the feature, A k B is an unlearnable adjacency matrix; k and C k It is A k The initialized learnable matrices, α and β, are used to balance B. k and C k Hyperparameters.

[0090] In one specific embodiment, the graph convolutional recognition network updates parameter B through backpropagation during training. k and C k .

[0091] In one specific embodiment, the data augmentation process for the unlabeled skeleton sequence to obtain an enhanced view specifically involves: processing the unlabeled skeleton sequence using data augmentation methods including shearing, cropping, flipping, and coordinate axis occlusion to obtain different enhanced views {x}. i |i=1,2}.

[0092] In one specific embodiment, the global loss, local loss, explicit and implicit losses all employ the InfoNCE loss function; the InfoNCE loss function is expressed as:

[0093]

[0094] Here, x and y are positive sample pairs from the same sample, and x and z are negative sample pairs from different samples; B(y) represents the set of all samples containing y in the current sample batch, and τ is a temperature hyperparameter used to balance the training process.

[0095] In one specific embodiment, such as Figure 3 As shown, the specific steps for constructing the global loss are as follows:

[0096] Enhance view {x iThe inputs of |i=1,2} to a global encoder sharing the same weights yield global features {z}. i |i=1,2};

[0097] global feature {z i |i=1,2} are processed by a predictor consisting of two fully connected layers to generate corresponding predicted features p1 and p2;

[0098] Since z1 and z2 come from the same original sample, it is inferred that p1 and p2 should contain similar feature information. Therefore, p1 is considered a positive sample pair of z2, p2 is considered a positive sample pair of z1, and the other samples are negative sample pairs.

[0099] Unlike traditional methods that require a large number of negative samples for comparative learning, this invention uses a prediction task to replace the comparison task between two views; by mapping global features to a prediction feature space, it establishes the prediction relationship between corresponding actions, maximizing the consistency between global features and prediction features, thereby avoiding the generation of trivial solutions to some extent; a global loss is constructed based on global features and prediction features.

[0100] L global =L Info (z1,p2)+L Info (z2,p1).

[0101] In one specific embodiment, such as Figure 4 As shown, the local loss is constructed, and the specific steps are as follows:

[0102] {x i |i=1,2} are divided into upper limb joints and lower limb joints

[0103] Each and The input is fed into the corresponding local encoder to obtain the local features of the upper limb. and local features of the lower limbs

[0104] Constructing local loss:

[0105]

[0106]

[0107] in, and These are positive sample pairs, {i,j} = {1,2}. This invention utilizes local features to minimize the distribution differences obtained by different data processing of the same human body parts.

[0108] In one specific embodiment, due to the Transformer's unique query-matching mechanism, its self-attention mechanism can estimate the correlation between features from multiple perspectives and mine the relationships between local features of actions at the feature level. The explicit and implicit losses are constructed based on salient and non-salient features, and the specific steps are as follows:

[0109] Employing the self-attention mechanism of Transformer, utilizing weight W Q W K W V z i Mapped to {Q,K,V}:

[0110] Q = z i W Q K = z i W K V=z i W V

[0111] The similarity between features is calculated using the Q and K matrices, then mapped to a low-dimensional space using V, and finally a salient region attention mask M is obtained through a linear layer and a sigmoid function. s Non-salient region attention mask M ns :

[0112]

[0113] M ns =IM s

[0114] Where, d k λ represents the feature dimension and is used for normalization; λ is a hyperparameter used to adjust the degree of fluctuation in feature changes during training.

[0115] Through M s and M ns Calculate the significance feature q s Non-significant features q ns :

[0116]

[0117]

[0118] The salient features q are processed through a multilayer perceptron (MLP) layer. s Non-significant features q ns Mapped to a lower-dimensional space;

[0119] In one specific embodiment, a sample is processed using two different data augmentation methods to obtain two views. These two views are then processed through corresponding Siamese networks to obtain features containing rich semantic information. Theoretically, the two views still represent the same sample, meaning the feature distributions should also be similar. Therefore, the salient features of the two views are used as positive sample pairs, and the global feature z is... i The non-significant features are used as latent negative sample pairs to constrain the model's learning ability, enabling it to learn reasonable action features, and constructing explicit and implicit losses:

[0120]

[0121]

[0122] in and Each is a positive sample pair, {i,j}={1,2}.

[0123] Furthermore, to maximize the consistency between similar features and minimize the distribution differences between features of the same category, a loss function for the entire network is obtained based on global loss, local loss, and explicit and implicit losses:

[0124] L total =L global +L up +L low +L s +L ns .

[0125] In this embodiment, the present invention proposes a human skeleton action recognition method based on self-supervised learning. Compared with traditional self-supervised action recognition methods that require a large number of negative samples, the present invention optimizes the memory consumption of the negative sample memory bank and uses a prediction task to replace the explicit positive and negative sample comparison task. Compared with traditional contrastive learning action recognition methods that only focus on global information, the present invention uses an adaptive action recognition algorithm with local feature-assisted contrastive learning. It utilizes local information from the overall skeleton and local explicit features of global features. The introduced adaptive spatiotemporal graph convolutional network effectively captures discriminative representations by learning the topology, and further enhances the learning ability by extracting salient and non-salient local features through an attention mask module.

[0126] Example 4

[0127] A self-supervised action recognition system based on local feature assistance includes a cascaded data acquisition module, a data augmentation module, a graph convolutional network module, a training module, and an action recognition module.

[0128] The data acquisition module is used to acquire unlabeled skeleton sequences;

[0129] The data augmentation module performs data augmentation processing on the unlabeled skeleton sequence to obtain an enhanced view;

[0130] The graph convolutional network module is used to obtain global features through a global encoder sharing the same weights, and input the global features into a predictor to obtain predicted features; a global loss is constructed based on the global features and predicted features; the augmented view is divided into upper limb joints and lower limb joints, and the upper limb joints and lower limb joints are input into the corresponding local encoders to obtain local features; a local loss is constructed based on the local features; salient and non-salient features are obtained based on the global features through a self-attention mechanism; explicit and implicit losses are constructed based on the salient and non-salient features.

[0131] The action recognition module is used to perform action recognition through a trained graph convolutional recognition network.

[0132] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A self-supervised action recognition method based on local feature assistance, characterized in that: The specific steps include the following: Obtain the unlabeled skeleton sequence; Data augmentation is performed on the unlabeled skeleton sequence to obtain an augmented view; The enhanced view is input into the graph convolutional recognition network; Global features are obtained by sharing the same weights in a global encoder, and these global features are then input into a predictor to obtain predicted features. A global loss is constructed based on the global features and the predicted features. The specific steps for constructing the global loss are as follows: Enhance the view Global features are obtained by inputting the data into a global encoder that shares the same weights. ; global features The predictor generates the corresponding predicted features after passing through a predictor consisting of two fully connected layers. and Will Seen as Positive sample pairs, Seen as Positive sample pairs are defined as positive sample pairs, and all other samples are defined as negative sample pairs. A prediction task replaces the task of comparing global features and predicted features; by mapping global features to the prediction feature space, the prediction relationship between corresponding actions is established to maximize the consistency between global features and predicted features; a global loss is constructed based on global features and predicted features. The enhanced view is divided into upper limb joints and lower limb joints. The upper limb joints and lower limb joints are then input into the corresponding local encoders to obtain local features. Construct a local loss based on local features; By employing a self-attention mechanism, salient and non-salient features are obtained based on global features; Construct explicit and implicit losses based on salient and insignificant features; A graph convolutional recognition network is trained based on global loss, local loss, and explicit and implicit losses. Action recognition is performed using a pre-trained graph convolutional recognition network.

2. The self-supervised action recognition method based on local feature assistance according to claim 1, characterized in that: The graph convolutional recognition network described above employs the spatiotemporally adaptive graph convolutional network STAGCN; specifically, STAGCN is represented as: in, Indicates features, It is an adjacency matrix that cannot be learned; and It is by The initial learnable matrices, α and β, are used for balancing. and Hyperparameters.

3. The self-supervised action recognition method based on local feature assistance according to claim 2, characterized in that: The graph convolutional recognition network updates its parameters through backpropagation during training. and .

4. The self-supervised action recognition method based on local feature assistance according to claim 1, characterized in that: The aforementioned data augmentation processing of the unlabeled skeleton sequence to obtain augmented views specifically involves: processing the unlabeled skeleton sequence using data augmentation methods including shearing, cropping, flipping, and coordinate axis occlusion to obtain different augmented views. .

5. The self-supervised action recognition method based on local feature assistance according to claim 4, characterized in that: Specifically, the global loss, local loss, explicit and implicit losses all employ the InfoNCE loss function; the InfoNCE loss function is expressed as: Here, x and y are positive sample pairs from the same sample, and x and z are negative sample pairs from different samples; B(y) represents the set of all samples containing y in the current sample batch, and τ is a temperature hyperparameter used to balance the training process.

6. The self-supervised action recognition method based on local feature assistance according to claim 5, characterized in that: The specific steps for constructing a local loss are as follows: Will Divided into upper limb joints and lower limb joints ; Each and The input is fed into the corresponding local encoder to obtain the local features of the upper limb. and local features of the lower limbs ; Constructing local loss: in, and Let {i,j} be positive sample pairs, and {i,j} = {1,2}.

7. The self-supervised action recognition method based on local feature assistance according to claim 6, characterized in that: By employing a self-attention mechanism, salient and non-salient features are obtained based on global features; The explicit and implicit losses are constructed based on salient and implicit features. The specific steps are as follows: Employing the self-attention mechanism of Transformer, utilizing weights Map global features to matrix: The similarity between features is calculated using the Q and K matrices, then mapped to a low-dimensional space using V, and finally, a salient region attention mask is obtained through a linear layer and a sigmoid function. Non-salient region attention mask : in, Represents the feature dimension, used for normalization; It is a hyperparameter used to adjust the degree of fluctuation in feature changes during training; pass and Calculate saliency features Non-significant features : The salient features are obtained through the multilayer perceptron (MLP) layer. Non-significant features Mapped to a lower-dimensional space; The salient features of the two views are used as positive sample pairs, and the global features are used as positive sample pairs. The non-significant features are used as latent negative sample pairs to construct explicit and implicit losses: in and Each is a positive sample pair, {i,j} = {1,2}.

8. The self-supervised action recognition method based on local feature assistance according to claim 7, characterized in that: When training a graph convolutional recognition network, the loss function is based on global loss, local loss, and explicit and implicit losses: 。 9. A self-supervised action recognition system based on local feature assistance, characterized in that: The method for implementing the self-supervised action recognition method as described in any one of claims 1 to 8 includes a cascaded data acquisition module, a data augmentation module, a graph convolutional network module, a training module, and an action recognition module. The data acquisition module is used to acquire unlabeled skeleton sequences; The data augmentation module performs data augmentation processing on the unlabeled skeleton sequence to obtain an enhanced view; The graph convolutional network module is used to obtain global features through a global encoder that shares the same weights, and then input the global features into a predictor to obtain predicted features. Construct a global loss based on global features and predicted features; The enhanced view is divided into upper limb joints and lower limb joints. The upper limb joints and lower limb joints are then input into the corresponding local encoders to obtain local features. Local loss is constructed based on local features; it is used to obtain salient and non-salient features based on global features through a self-attention mechanism; a graph convolutional recognition network is trained based on global loss, local loss, and explicit and implicit losses. The action recognition module is used to perform action recognition through a trained graph convolutional recognition network.

Images