Few-sample skeleton action recognition method and system based on double attention focus network

Abstract

The invention discloses a few-sample skeleton action recognition method and system based on a double-attention focus network, and relates to the technical field of action recognition, and the method comprises the steps: inputting skeleton data into a pre-established double-attention focus network model DAFNet, carrying out the spatial-temporal feature extraction through a spatial-temporal diagram convolution network, and obtaining a spatial-temporal feature extraction result; outputting to obtain support sample features and query sample features; performing dynamic optimization adjustment based on a spatial self-attention mechanism and a cross attention mechanism to obtain optimized support sample features and query sample features; calculating the similarity between the support sample features and the query sample features based on the optimized support sample features and query sample features, redistributing sample weights based on the similarity, and obtaining the redistributed support sample features and query sample features based on the sample weights; and respectively calculating a frame-level distance and carrying out dynamic planning to obtain a total distance between the support set and the query set, and taking the total distance as a skeleton action recognition result.

Description

Technical Field

[0001] This invention relates to the field of action recognition technology, specifically a few-sample skeleton action recognition method and system based on a dual-attention focus network. Background Technology

[0002] In recent years, Human Action Recognition (HAR) has become a key research direction in the field of computer vision, showing broad prospects in many applications such as intelligent surveillance, healthcare, sports analysis, and human-computer interaction. Deep learning-based methods have achieved significant breakthroughs on large-scale labeled datasets; however, these methods heavily rely on large amounts of manually labeled training data. In real-world scenarios, data acquisition is often limited by high collection costs and the complexity of professional annotation. This problem is particularly prominent in professional fields such as medicine. For example, collecting and annotating medical action videos often takes years and requires the collaboration of multiple experts, severely limiting the scalability and generalization ability of the models.

[0003] To address the problem of data scarcity, Few-Shot Action Recognition (FSAR) has received increasing attention in recent years. Its goal is to identify unlabeled actions (i.e., query samples) of new categories when only a small number of labeled samples exist for each category. Compared to traditional supervised learning methods, few-shot learning does not rely on large-scale datasets, significantly reducing data labeling costs and thus improving the model's applicability in real-world applications. However, neither single intra-skeleton modeling nor inter-skeleton matching can fully characterize human action features. Summary of the Invention

[0004] To address the shortcomings mentioned in the background section, the present invention aims to provide a few-sample skeleton action recognition method and system based on a dual attention focus network.

[0005] Firstly, the objective of this invention can be achieved through the following technical solution: a few-sample skeleton action recognition method based on a dual-attention focus network, the method comprising the following steps:

[0006] The skeleton data is obtained and input into the pre-established dual attention focusing network model DAFNet. Spatiotemporal features are extracted through a spatiotemporal graph convolutional network, and the support sample features and query sample features are output. The support sample features and query sample features are dynamically optimized and adjusted based on spatial self-attention mechanism and cross-attention mechanism to obtain optimized support sample features and query sample features.

[0007] The similarity between the support sample features and query sample features is calculated based on the optimized support sample features and query sample features. The sample weights are then reassigned based on the similarity between the support sample features and query sample features. The reassigned support sample features and query sample features are obtained based on the sample weights.

[0008] The frame-level distance between the support sample features and query sample features is calculated based on the reallocated support sample features and query sample features. The frame-level distance between the support sample features and query sample features is dynamically planned based on the Dynamic Time Warping (DWT) algorithm to obtain the total distance between the support set and the query set, which is used as the skeleton action recognition result.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the optimization process of the spatial self-attention mechanism is as follows:

[0010] In DAFNet, spatial self-attention is applied to the query set features H q and support set features H s The transformation formula is shown below:

[0011]

[0012] in The transformation matrix supports the following feature transformations:

[0013]

[0014] In the formula, This indicates that the query sample features H q ; This indicates that for the supporting sample feature H s The linear transformation matrix; d represents the feature dimension, used to scale the dot product result; SoftMax represents the standard softmax function, used to calculate the attention weights; Let represent the feature representations after attention fusion of the query set and support set, respectively; where: This represents the original feature representation of the skeleton sequence of the query set, where C is the number of channels, T is the number of time frames, and J is the number of key points; This represents the original feature representation of the support set skeleton sequence; This represents the query set feature transformation matrix, used to map the original features to the attention space; · represents the characteristic transformation matrix of the support set; · represents matrix multiplication.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: an optimization process for the cross-attention mechanism:

[0016] Based on the spatial self-attention output, prepare the feature vectors of the query set and support set. A linear transformation is then performed to make it suitable for cross-attention calculation, as shown in the following equation:

[0017]

[0018] in Represents the weight matrix;

[0019] The dot product of the query vector and the key vector is normalized using the Softmax function, and the attention weights in the cross space are calculated. These weighted value vectors V are then combined to form the final cross-feature representation, as shown in the following formula:

[0020]

[0021]

[0022] In the formula, Q q Let K represent the query vector of the query set, K represent the feature representation to be matched in the current action segment, and K represent the feature representation to be matched in the current action segment. s V represents the key vector of the support set, and V represents the semantic anchoring features of the reference action segment. q V represents the value vector of the query set. s d represents the value vector of the support set, and d represents the scaling factor.

[0023] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating the similarity between the supporting sample features and the query sample features, comprising:

[0024] The calculation formula is based on cosine similarity and is defined as follows:

[0025]

[0026] in, Let represent the feature vectors of the supporting sample and the query sample, respectively. · represents the dot product of the two vectors, and ||·|| represents the L2 norm of the vectors. The entire denominator... It represents the product of the magnitudes of two vectors.

[0027] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of reallocating sample weights based on the similarity between supporting sample features and query sample features, comprising:

[0028] Based on the similarity index, a weight coefficient γ is defined for each supporting sample. i The calculation method is as follows:

[0029]

[0030] The combined prototype C for each category k k It is obtained by weighting and aggregating its supporting samples, and the formula is:

[0031]

[0032] In the formula, γ i Indicates support for sample G i s Relative to query sample G i q The weight coefficients; Sim(·,·) represents the similarity function between feature vectors; S k C represents the set of supporting samples corresponding to the k-th class; k f represents the combinatorial prototype of the k-th class; φ (G j s ) indicates support for sample G j s Image feature representation.

[0033] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the process of calculating the frame-level distance between the support sample features and the query sample features based on the reassigned support sample features and query sample features, comprising:

[0034] Based on support sample feature sequences With query sample feature sequence The frame-level distance between them is calculated using the following formula:

[0035]

[0036] Among them, D g (l,m) represents the frame-level distance between the l-th frame of the support set and the m-th frame of the query set;

[0037] Construct a distance matrix E, where each element E l,m This represents the frame-level distance between frame l and frame m. After obtaining the distance matrix, the cumulative distance Ω(l,m) is calculated using the DTW algorithm through dynamic programming, and its recursive formula is as follows:

[0038] Ω(l,m)=E(l,m)+min{Ω(l-1,m-1),Ω(l-1,m),Ω(l,m-1)}(10)

[0039] Where Ω(l,m) represents the minimum cumulative distance from the l-th frame of the query sequence to the m-th frame of the support sequence;

[0040] Through dynamic programming, the total distance between the support set and the query set can be defined as:

[0041] d(Q,C k )=Ω(m,m)(11)d(Q,C k ) represents the support set feature sequence C k The DTW distance between the query set feature sequence Q and the query set feature sequence Q is used to measure the similarity between two action samples.

[0042] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: the optimization objective function L of the pre-established dual attention focusing network model DAFNet. match as follows:

[0043]

[0044] Where, N q Indicates the number of query samples. and y i Representing samples respectively The predicted labels and the actual labels.

[0045] Secondly, in order to achieve the above objectives, this invention discloses a few-shot skeleton action recognition system based on a dual-attention focus network, comprising:

[0046] The feature processing module is used to acquire skeleton data, input the skeleton data into the pre-built dual attention focusing network model DAFNet, extract spatiotemporal features through spatiotemporal graph convolutional network, and output support sample features and query sample features. Based on spatial self-attention mechanism and cross attention mechanism, the support sample features and query sample features are dynamically optimized and adjusted to obtain optimized support sample features and query sample features.

[0047] The weight allocation module is used to calculate the similarity between the support sample features and the query sample features based on the optimized support sample features and query sample features, reallocate the sample weights based on the similarity between the support sample features and the query sample features, and obtain the reallocated support sample features and query sample features based on the sample weights.

[0048] The action recognition module is used to calculate the frame-level distance between the support sample features and the query sample features based on the reassigned support sample features and query sample features. It uses the Dynamic Time Warping (DWT) algorithm to dynamically plan the frame-level distance between the support sample features and the query sample features, and obtains the total distance between the support set and the query set as the skeleton action recognition result.

[0049] In another aspect of the present invention, in order to achieve the above-mentioned objective, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The memory stores the computer program capable of running on the processor. When the processor loads and executes the computer program, it employs the few-sample skeleton action recognition method based on a dual attention focus network as described above.

[0050] In another aspect of the present invention, in order to achieve the above objective, a computer-readable storage medium is disclosed, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is loaded and executed by a processor, it employs the few-sample skeleton action recognition method based on a dual attention focus network as described above.

[0051] The beneficial effects of this invention are:

[0052] This invention utilizes a focused attention mechanism to enhance the discriminative feature learning ability in few-shot skeleton action recognition (FSAR). Combined with an adaptive sample weighting strategy, it can effectively improve action classification performance. The adaptive sample weighting strategy based on cosine similarity improves the class discrimination ability, while the dynamic time warping (DTW) method effectively optimizes the time alignment process, enhances the model's generalization ability in few-shot scenarios, and promotes the continuous improvement of few-shot action recognition performance. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0055] Figure 2 This is a schematic diagram of the overall structure of the DAFNet of this invention;

[0056] Figure 3 This is a schematic diagram of the 5-way-N-shot experimental results using the ST-GCN backbone network on the NTU-T dataset of this invention;

[0057] Figure 4 This is a schematic diagram comparing the performance of the baseline method and the method of the present invention under the NTU-T dataset 1-shot setting in an embodiment of the present invention;

[0058] Figure 5This is a visualization diagram of the feature t-SNE under the N-way 1-shot setting on the NTU-T dataset of this invention;

[0059] Figure 6 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Example 1:

[0062] like Figure 1 As shown, a few-shot skeleton action recognition method based on a dual-attention focus network includes the following steps:

[0063] S101: Obtain skeleton data, input the skeleton data into the pre-established dual attention focusing network model DAFNet, extract spatiotemporal features through spatiotemporal graph convolutional network, and output support sample features and query sample features; dynamically optimize and adjust the support sample features and query sample features based on spatial self-attention mechanism and cross-attention mechanism to obtain optimized support sample features and query sample features.

[0064] The optimization process of the spatial self-attention mechanism is as follows:

[0065] The dynamic changes and spatial relationships of different joints often contain rich motion information, but graph convolution operations alone are often insufficient to fully capture these details. Graph convolution mainly focuses on feature extraction from local neighborhoods, which has limitations in handling complex global relationships between joints. The joints within each frame may contain many untapped motion semantic features. To comprehensively mine the semantic information within the skeleton, this invention proposes a spatial self-attention mechanism that enhances the representational power of joint features by calculating the correlation weights between joints, thereby capturing more global semantic features.

[0066] This method not only considers the relationships between adjacent joints but also establishes connections between joints on a larger scale, effectively enhancing feature representation capabilities. In DAFNet, spatial self-attention is applied to the query set features H. q and support set features H s The transformation formula is shown below:

[0067]

[0068] in The transformation matrix supports the following feature transformations:

[0069]

[0070] In the formula, This indicates that the query sample features H q ; This indicates that for the supporting sample feature H s The linear transformation matrix; d represents the feature dimension, used to scale the dot product result; SoftMax represents the standard softmax function, used to calculate the attention weights; These represent the feature representations after the attention fusion of the final query set and support set, respectively.

[0071] The optimization process of the cross-attention mechanism: Based on spatial self-attention, this invention further introduces a spatial cross-attention mechanism. Traditional methods often rely on Euclidean distance between joint features for matching, but this approach often fails to highlight the subtle differences that are crucial in action recognition. For example, when classifying visually similar actions such as "brushing teeth" and "washing face," the subtle differences in hand and elbow positions are particularly critical. The interaction between different skeleton sequences can amplify these subtle differences, thereby enhancing the model's ability to distinguish action features and improving the matching effect between the query set and the support set.

[0072] This mechanism optimizes spatial relationship modeling by comparing joint features in two skeleton graphs. Its core task is to dynamically compare and fuse features in the query graph and support graph, thereby identifying and highlighting key similarities and differences between them. The specific steps are as follows: First, based on the spatial self-attention output, feature vectors for the query set and support set are prepared. Then, a linear transformation is performed on it to make it suitable for cross-attention calculation, as shown in the following equation:

[0073] Based on the spatial self-attention output, prepare the feature vectors of the query set and support set. A linear transformation is then performed to make it suitable for cross-attention calculation, as shown in the following equation:

[0074]

[0075] in Represents the weight matrix;

[0076] The Softmax function is used to normalize the dot product of the query vector and the key vector, and attention weights in the cross space are calculated to highlight the matching relationship between the support set and the query set action features. The weighted value vectors V are then combined to form the final cross feature representation, revealing subtle differences and key commonalities between actions. The specific formula is as follows:

[0077]

[0078]

[0079] In the formula, Q q K represents the query vector of the query sample; s V represents the key vector of the supporting samples; q V s These represent the value vectors of the query sample and the supporting samples, respectively.

[0080] Focused attention is a comprehensive mechanism that combines self-attention and cross-attention to enhance feature extraction capabilities in skeleton action recognition. Self-attention strengthens the representation of joints within the skeleton, while cross-attention enhances the interaction modeling capabilities between different skeleton sequences. The combination ensures the model focuses on the most discriminative and semantically relevant features, thereby significantly improving the accuracy of action recognition.

[0081] S102: Calculate the similarity between the support sample features and the query sample features based on the optimized support sample features and query sample features; reallocate the sample weights based on the similarity between the support sample features and the query sample features; and obtain the reallocated support sample features and query sample features based on the sample weights.

[0082] The process of calculating the similarity between the supporting sample features and the query sample features includes:

[0083] Optimized Prototype Network (OP-N) aims to enhance the robustness of networks and improve their resistance to noise in classification tasks. Its core is the Sample Efficacy Optimization (SEO) method. This method addresses the inherent shortcomings of traditional prototype networks, which typically use simple arithmetic averaging to construct class prototypes. Traditional methods perform poorly with noisy data or class imbalance, often generating class prototypes that fail to accurately reflect class characteristics and are susceptible to outliers. To solve these problems, OP-N introduces a novel prototype construction paradigm: using cosine similarity instead of the mean generation mechanism to achieve more robust class representations. By analyzing the cosine similarity between query samples and support samples, OP-N can generate more representative prototypes and effectively improve resistance to noise and data bias. To quantify the relationship between samples, OP-N proposes a new metric: the Similarity Index (Sim), calculated based on cosine similarity, defined as follows:

[0084]

[0085] in, These represent the feature vectors of the supporting sample and the query sample, respectively. The larger the Sim value, the more similar the two samples are, and correspondingly, the supporting sample should have a greater weight in prototype construction. These represent supporting samples. and query samples The modulus of the feature vectors; the product of the modulus in the denominator is used to normalize the feature dot product result and calculate the cosine similarity, with a value range between [-1, 1].

[0086] The process of reallocating sample weights based on the similarity between supporting sample features and query sample features includes:

[0087] Based on the similarity index, a weight coefficient γ is defined for each supporting sample. i This is used to adjust its contribution to the category prototype, and is calculated as follows:

[0088]

[0089] The combined prototype C for each category k k It is obtained by weighting and aggregating its supporting samples, and the formula is:

[0090]

[0091] In the formula, γ i Indicates supporting samples For the weighting coefficients used to generate the combined prototype; S k Let represent the set of all supporting samples in the k-th class; To support the sample Feature representation;

[0092] This combined prototype can more accurately capture the central tendency of categories in the feature space, improve the model's ability to represent category characteristics, and is also more conducive to handling samples with complex pattern distributions.

[0093] This optimized prototyping approach not only improves the sensitivity of class prototypes to fine-grained differences but also enhances the model's discriminative ability in the face of noise and imbalanced data. Through the SEO module, OP-N effectively integrates the contributions of each supporting sample to the class prototype, thereby generating a more representative and robust class-center representation.

[0094] S103: Calculate the frame-level distance between the support sample features and the query sample features based on the reallocated support sample features and query sample features. Dynamically plan the frame-level distance between the support sample features and the query sample features based on the Dynamic Time Warping (DWT) algorithm to obtain the total distance between the support set and the query set, which is used as the skeleton action recognition result.

[0095] The process of calculating the frame-level distance between the support sample features and the query sample features based on the reallocated support sample features includes:

[0096] Due to different start and end times of acquisition, different skeleton sequences may have inconsistent lengths. Even for the same action, differences in execution speed can lead to different frame counts and time intervals. To accurately compare skeleton sequences with different lengths and speeds, flexible matching in the time dimension is necessary, and a dynamic alignment method has been introduced to address this issue. Dynamic Time Warping (DTW) is a common time series alignment method [35, 36] that can dynamically adjust the matching relationship between sequences to find the optimal alignment path. DTW can achieve dynamic alignment while maintaining the temporal order of the sequences. Specifically, it first calculates the support set feature sequence. With query set feature sequence The frame-level distance between them is calculated using the following formula:

[0097]

[0098] Among them, D g (l,m) represents the frame-level distance between the l-th frame of the support set and the m-th frame of the query set;

[0099] Construct a distance matrix E, where each element E l,m This represents the frame-level distance between frame l and frame m. After obtaining the distance matrix, the cumulative distance Ω(l,m) is calculated using the DTW algorithm through dynamic programming, and its recursive formula is as follows:

[0100] Ω(l,m)=E(l,m)+min{Ω(l-1,m-1),Ω(l-1,m),Ω(l,m-1)} (10)

[0101] Where Ω(l,m) represents the minimum cumulative distance from the l-th frame of the query sequence to the m-th frame of the support sequence; E∈R L×M To support the distance matrix between the sample and the query sample frame sequence.

[0102] Through dynamic programming, the total distance between the support set and the query set can be defined as:

[0103] d(Q,C k )=Ω(m,m)(11). d(Q,C k ) indicates support set C k The overall distance between the query set Q and the query set Q is used for combination matching and classification.

[0104] The optimization objective function L of the pre-established dual attention focusing network model DAFNet match as follows:

[0105]

[0106] Where, N q Indicates the number of query samples. and y i Representing samples respectively The predicted and true labels are compared. The loss function consists of a matching loss and a dispersion regularization term. The former ensures that the model accurately identifies the action category, while the latter is used to adjust the discriminativeness of the feature representation, thereby improving the model's generalization ability and robustness.

[0107] Specifically, the present invention will be further illustrated below through embodiments:

[0108] Dataset

[0109] The NTU RGB+D 120 dataset is an important resource for researching human motion recognition based on skeleton data. This dataset contains 114,480 skeleton sequences, recording the coordinates of 25 key points in 3D space, covering 120 different motion types. These motion categories are mainly divided into three classes: 82 categories of daily behavioral actions, 12 categories of health-related actions, and 26 categories of two-person interactive actions.

[0110] To facilitate experimental analysis, the dataset was divided into two subsets, named "NTU-S" and "NTU-T". "NTU-S" randomly samples 60 samples from each class, while "NTU-T" selects 30 samples from each class. All samples are divided as follows: 80 classes for training, 20 classes for validation, and 20 classes for testing, providing a structured experimental framework for model performance evaluation.

[0111] The Kinetics dataset was collected by the DeepMind team and focuses on the recognition and analysis of human motion videos. The dataset contains more than 260,000 video clips, covering 400 different human motion categories, with about 400 video instances for each category. The raw data exists in the form of complete videos. To adapt to the research scenario of skeleton recognition, the researchers used OpenPose technology

[39] to estimate the human pose in the video, thereby extracting skeleton information.

[0112] During skeleton extraction, 18 key skeletal nodes were identified and used to construct a graph representation. Each node's features include its horizontal and vertical distances to the top-left corner of the video, as well as a confidence score provided by OpenPose. To ensure research efficiency, only the top 120 action categories were selected, and 100 samples from each category were chosen for analysis. The training, validation, and testing partitions were consistent with the NTU RGB+D 120 dataset to facilitate result comparison and generalization ability analysis.

[0113] Baseline method

[0114] ProtoNet is based on a prototype network architecture and uses Euclidean distance as a metric for human action classification. This method first extracts features using a spatiotemporal encoder, and then completes the classification based on the distance from the sample to the prototype class in the feature space.

[0115] Dynamic Time Warping (DTW) is a method for determining the optimal matching relationship between two time series. It uses a dynamic programming algorithm to align the sequences and is one of the important tools in video processing tasks such as few-sample action recognition.

[0116] PairNorm is a normalization layer for Graph Neural Networks (GNNs) designed to prevent excessive smoothing of node features. Its core idea is to rescale the node representations using a constant hyperparameter to preserve the diversity between the original node features. PairNorm has two key objectives: providing a constant divergence metric (i.e., a constant hyperparameter) and model independence (allowing it to be used as a general layer in various GNN models). However, it's worth noting that the selection of this constant hyperparameter often depends on prior knowledge of the input graph data, which can be difficult to determine in practice. The essence of PairNorm lies in controlling the divergence of features between nodes, thereby avoiding the problem of all node embeddings tending to be consistent.

[0117] ESR-MM proposes an Enhanced Skeleton Representation and Multi-relational Metrics framework to address the challenges of feature representation and class discrimination in few-sample skeleton action recognition. This method introduces Frobenius Norm Diversity Loss (FNDL) to mitigate the oversmoothing problem in graph convolutional networks, improving feature diversity and discriminative power. It also incorporates Supportive Adaptive Attention (SAA) for task-specific feature optimization, employs Bi-Temporally Consistent Hausdorff Distance (Bi-TCMHM) to enhance temporal alignment, and utilizes Prototype Discriminative Contrast Loss (PDCL) to enhance class separability.

[0118] DASTM is one of the baseline variants implemented in this invention. This model precisely aligns the spatial structure by introducing a skeleton space activation mechanism and combines it with the DTW method for time synchronization. Compared to the traditional DTW architecture, DASTM is more refined in identifying salient key nodes in the skeleton graph and can construct a richer graph structure representation space, thereby effectively improving the ability to express and discriminate human actions.

[0119] The method proposed in this invention is consistent with the above baseline in terms of modeling settings, does not rely on pre-trained models, and does not introduce additional auxiliary datasets, thus maintaining a fair comparative benchmark in few-shot learning tasks.

[0120] Experimental details

[0121] This model employs a task-driven training method. In each training round, several action categories are randomly selected, and a support set consisting of a fixed number of labeled samples is constructed for each category. Subsequently, samples are randomly drawn from the remaining categories to form the query set, ensuring that the training task can cover diverse situations that may be encountered in real-world applications. The data preprocessing follows a unified strategy: for the NTU-S and NTU-T datasets, thirty frames are randomly sampled from each skeleton sequence; for the Kinetics dataset, fifty frames are sampled from each video sequence as input, ensuring that the input lengths of the support set and query set are consistent. The model skeleton encoder mainly adopts three structures: spatiotemporal graph convolutional network, two-stream adaptive graph convolutional network, and multi-scale... Figure 3 A 3D convolutional network was used. During training, the Adam optimizer was employed with an initial learning rate of 0.001 and a learning rate decay strategy: if the validation set accuracy showed no improvement for ten consecutive epochs, the learning rate was automatically halved to promote convergence. Furthermore, the model incorporated rank maximization to optimize the skeleton feature matrix, thereby improving its ability to discriminate skeleton graph structures. Each training epoch contained 500 sub-tasks: five-class sample tasks and five-class five-sample tasks. Five action classes were randomly selected from the dataset for training each time. Training was performed on an NVIDIA GeForce RTX 4090 graphics card, with each experiment repeated three times to report the average accuracy.

[0122] Table 1. 5-way 1-shot benchmark performance on NTU-S, NTU-T, and Kinetics datasets, highlighting the highest accuracy (%).

[0123]

[0124] In a 5-class 1-sample experimental setup, this invention demonstrated significant performance improvements across all metrics. On the NTU-S dataset, combined with the 2s-AGCN skeleton structure, the accuracy of this invention reached 77.1%, a 3.1 percentage point improvement over DASTM's 74.0%. To further verify the method's versatility and robustness, this study conducted systematic tests on three mainstream datasets (NTU-S, NTU-T, and Kinetics) and three different skeleton coding structures (ST-GCN, 2s-AGCN, and MS-G3D). Despite significant background noise in the Kinetics dataset, this invention still exhibited good performance. In a 5-class 1-sample setup, combined with the MS-GCN structure, it achieved an accuracy of 79.1% on the NTU-S dataset, surpassing DASTM's 76.3%, a 2.8% improvement. Regardless of the skeleton structure used, this invention consistently outperformed the comparative methods in terms of accuracy and stability, and is adaptable to various structures ranging from the simple ST-GCN to the complex 2s-AGCN and MS-G3D. In summary, this invention demonstrates excellent performance on both the noisy Kinetics dataset and the complex NTU-S dataset, significantly outperforming the baseline method DASTM, thus verifying its wide applicability and strong robustness.

[0125] Tables 2 and 3 illustrate the detailed impact analysis of each component on the model's learning performance. The baseline model proposed in this invention integrates a cross-attention mechanism and a dynamic temporal warping (DTW) method. Here, SSA represents spatial self-attention, SEO represents sample performance optimization, and SCA represents spatial cross-attention. The baseline method calculates the difference between two skeleton sequences in the overall dimension using the Frobenius norm to construct the inter-frame distance matrix Elm. In contrast, the method of this invention uses cosine similarity to calculate the similarity between local feature vectors of the skeleton sequences, more effectively capturing directional differences at specific moments and joints; this method is called Cosine Similarity-Based Frame Distance (CSFD).

[0126] In the 5-way 1-shot task, the effectiveness of sample efficiency optimization is limited because there is only one sample per class. However, in the 5-way 5-shot task, the method proposed in this invention shows greater potential. In both settings, the method of this invention outperforms the baseline.

[0127] In a 5-way-1-shot setting, using the ST-GCN backbone network, the accuracy of the model on the NTU-T dataset improved from 75.1% to 76.6% after adding SSA and SEO, and further improved to 77.6% after adding CSFD. Similar improvements were observed in the 2s-AGCN and MS-G3D backbone networks. In a 5-way-5-shot setting, the method of this invention significantly outperformed the baseline on all backbone networks. For example, on ST-GCN, the accuracy on NTU-T improved from 83.0% to 87.3% after adding SSA and SEO, and further improved to 87.8% after adding CSFD; on the NTU-S dataset, the accuracy improved from 86.8% to 89.8% using the 2s-AGCN backbone network.

[0128] To further verify the effectiveness of the proposed method, the model was evaluated under different sample sizes. For example... Figure 3 As shown, in the 5-way-N-shot learning task, the method of the present invention outperforms the baseline in all shot number settings, and achieves the highest accuracy of 90.6% in 10-shot. Figure 4 The invention demonstrates significant improvements in action categories such as "combing hair," "jumping," "applying hand cream," and "waving both hands." For actions requiring multi-part coordination or significant temporal variations (such as jumping), the spatial self-attention mechanism can capture the complex relationships between different parts of the action; for categories relying on local joint movements (such as applying hand cream), the spatial cross-attention mechanism enhances the ability to distinguish key joint features.

[0129] The model of this invention first utilizes a self-attention mechanism to explore spatial relationships within a sequence, thereby strengthening key interactions within the sequence, and then uses a cross-attention mechanism to highlight the interrelationships between sequences, thus achieving superior performance. This multi-layered attention mechanism, known as "focused attention," enables the model to more accurately identify and classify complex action categories, achieving good generalization across multiple instances. Results show that the proposed method has broad applicability and a stable performance improvement.

[0130] Table 2 shows a small-sample ablation experiment based on 5-way-1-shot motion recognition.

[0131]

[0132] Table 3 shows a small sample ablation experiment based on 5-way-5-shot motion recognition.

[0133]

[0134]

[0135] To further clarify the role of the attention mechanism, this invention visualizes the feature responses. For example... Figure 4 This study demonstrates the challenges in recognizing small movements with only minor joint changes, which are prone to misclassification in few-shot learning scenarios. Specifically, the model struggles to distinguish between "writing," "nail clipping," and "chest pain" because these movements involve only subtle local hand movements with minimal dynamic changes. "Writing" and "nail clipping" both involve slight rotation of the fingertips and wrist, while "chest pain" manifests as the palm pressing against the chest. The similarity in hand posture and movement trajectory further complicates classification. Error visualization results show that the model focuses excessively on the palm and wrist area, neglecting the grasping motion in "nail clipping" and the pressing process in "chest pain," leading to reduced class separability. This is particularly evident in the 1-shot setting, where the model lacks sufficient temporal information to capture these fine-grained differences at the beginning and end of the movement. Furthermore, in the early stages of the movement, "back kick" and "side kick" also exhibit small changes in the knee and hip joints, making it difficult for the model to extract key dynamic features and ultimately leading to misclassification as other leg-related movements.

[0136] Figure 5 The visualization of feature embedding before and after applying the proposed spatial attention and matching strategy is further demonstrated. In few-shot learning tasks, due to the limited number of training samples, models often struggle to form clear feature clusters. After applying the proposed attention and matching strategy to the initial features, the distribution of query and support samples becomes more compact, thus significantly improving class separability. Figure 5 (a) and Figure 5 (c) demonstrates the clustering performance on a 5-way task with 5 classes, where the feature embeddings exhibit more significant aggregation and clearer inter-class boundaries. On a 10-way task with 10 classes, the proposed method also effectively enhances the discriminative power between classes, validating its effectiveness and advantages in few-shot learning scenarios.

[0137] In basic t-SNE (t-distributed random neighbor embedding) dimensionality reduction visualization, due to the scarcity of samples, the original feature embeddings in the graph appear quite scattered, making it difficult to form clear category clusters. However... Figure 5 (b) and Figure 5 (d) shows the visualization effect of adaptive t-SNE after applying spatial attention mechanism and matching strategy.

[0138] Example 2: To achieve the above objective, such as Figure 6As shown, based on Embodiment 1, this invention discloses a few-shot skeleton action recognition system based on a dual-attention focus network, comprising:

[0139] Feature processing module 11 is used to acquire skeleton data, input the skeleton data into the pre-established dual attention focusing network model DAFNet, extract spatiotemporal features through spatiotemporal graph convolutional network, and output support sample features and query sample features; based on spatial self-attention mechanism and cross attention mechanism, the support sample features and query sample features are dynamically optimized and adjusted to obtain optimized support sample features and query sample features;

[0140] The weight allocation module 12 is used to calculate the similarity between the support sample features and the query sample features based on the optimized support sample features and query sample features, reallocate the sample weights based on the similarity between the support sample features and the query sample features, and obtain the reallocated support sample features and query sample features based on the sample weights.

[0141] The action recognition module 13 is used to calculate the frame-level distance between the support sample features and the query sample features based on the reassigned support sample features and query sample features, and to dynamically plan the frame-level distance between the support sample features and the query sample features based on the Dynamic Time Warping (DWT) algorithm to obtain the total distance between the support set and the query set, which is used as the skeleton action recognition result.

[0142] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0143] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0144] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0145] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A few-shot skeleton action recognition method based on a dual-attention focus network, characterized in that, The method includes the following steps: The skeleton data is obtained and input into the pre-established dual attention focusing network model DAFNet. Spatiotemporal features are extracted through a spatiotemporal graph convolutional network, and the support sample features and query sample features are output. The support sample features and query sample features are dynamically optimized and adjusted based on spatial self-attention mechanism and cross-attention mechanism to obtain optimized support sample features and query sample features. The similarity between the support sample features and query sample features is calculated based on the optimized support sample features and query sample features. The sample weights are then reassigned based on the similarity between the support sample features and query sample features. The reassigned support sample features and query sample features are obtained based on the sample weights. The frame-level distance between the support sample features and query sample features is calculated based on the reallocated support sample features and query sample features. The frame-level distance between the support sample features and query sample features is dynamically planned based on the Dynamic Time Warping (DWT) algorithm to obtain the total distance between the support set and the query set, which is used as the skeleton action recognition result.

2. The few-shot skeleton action recognition method based on a dual-attention focus network according to claim 1, characterized in that, The optimization process of the spatial self-attention mechanism is as follows: In DAFNet, spatial self-attention is applied to the query set features H q and support set features H s The transformation formula is shown below: in The transformation matrix supports the following feature transformations: Wherein: H q ∈R C×T×J This represents the original feature representation of the skeleton sequence of the query set, where C is the number of channels, T is the number of time frames, and J is the number of key points; This represents the original feature representation of the support set skeleton sequence; This represents the query set feature transformation matrix, used to map the original features to the attention space; · represents the characteristic transformation matrix of the support set; · represents matrix multiplication.

3. The few-shot skeleton action recognition method based on a dual-attention focus network according to claim 1, characterized in that, The optimization process of the cross-attention mechanism: Based on the spatial self-attention output, prepare the feature vectors of the query set and support set. A linear transformation is then performed to make it suitable for cross-attention calculation, as shown in the following equation: in Represents the weight matrix; The dot product of the query vector and the key vector is normalized using the Softmax function, and the attention weights in the cross space are calculated. These weighted value vectors V are then combined to form the final cross-feature representation, as shown in the following formula: Among them, Q q Let K represent the query vector of the query set, K represent the feature representation to be matched in the current action segment, and K represent the feature representation to be matched in the current action segment. s V represents the key vector of the support set, and V represents the semantic anchoring features of the reference action segment. q V represents the value vector of the query set. s d represents the value vector of the support set, and d represents the scaling factor.

4. The few-shot skeleton action recognition method based on a dual-attention focus network according to claim 1, characterized in that, The process of calculating the similarity between the supporting sample features and the query sample features includes: The calculation formula is based on cosine similarity and is defined as follows: in, Let represent the feature vectors of the supporting sample and the query sample, respectively. · represents the dot product of the two vectors, and ||·|| represents the L2 norm of the vectors. The entire denominator... It represents the product of the magnitudes of two vectors.

5. The few-shot skeleton action recognition method based on a dual-attention focus network according to claim 1, characterized in that, The process of reallocating sample weights based on the similarity between supporting sample features and query sample features includes: Based on the similarity index, a weight coefficient γ is defined for each supporting sample. i The calculation method is as follows: The combined prototype C for each category k k It is obtained by weighting and aggregating its supporting samples, and the formula is: In the formula, γ i Indicates supporting samples relative to the query sample The weight coefficients; Sim(·,·) represents the similarity function between feature vectors; S k C represents the set of supporting samples corresponding to the k-th class; k Represents the composite prototype of the k-th class; Indicates supporting samples Image feature representation.

6. The few-shot skeleton action recognition method based on a dual-attention focus network according to claim 1, characterized in that, The process of calculating the frame-level distance between the support sample features and the query sample features based on the reallocated support sample features includes: Based on support sample feature sequences With query sample feature sequence The frame-level distance between them is calculated using the following formula: Among them, D g (l,m) represents the frame-level distance between the l-th frame of the support set and the m-th frame of the query set; Construct a distance matrix E, where each element E l,m That is, the frame-level distance between the l-th frame and the m-th frame. This represents the feature vector of the l-th frame in the support set; Let represent the feature vector of the m-th frame in the query set; after obtaining the distance matrix, the cumulative distance Ω(l,m) is calculated using the DTW algorithm through dynamic programming, and its recursive formula is as follows: Ω(l,m)=E(l,m)+min{Ω(l-1,m-1),Ω(l-1,m),Ω(l,m-1)}(10)d(Q,C k ) Where Ω(l,m) represents the minimum cumulative distance from the l-th frame of the query sequence to the m-th frame of the support sequence; Through dynamic programming, the total distance between the support set and the query set can be defined as: d(Q,C k )=Ω(m,m)(11) d(Q,C k ) represents the support set feature sequence C k The DTW distance between the query set feature sequence Q and the query set feature sequence Q is used to measure the similarity between two action samples.

7. The few-shot skeleton action recognition method based on a dual-attention focus network according to claim 1, characterized in that, The optimization objective function L of the pre-established dual attention focusing network model DAFNet match as follows: Where, N q Indicates the number of query samples. and y i Representing samples respectively The predicted labels and the actual labels.

8. A few-shot skeleton action recognition system based on a dual-attention focus network, characterized in that, include: The feature processing module is used to acquire skeleton data, input the skeleton data into the pre-built dual attention focusing network model DAFNet, extract spatiotemporal features through spatiotemporal graph convolutional network, and output support sample features and query sample features. Based on spatial self-attention mechanism and cross attention mechanism, the support sample features and query sample features are dynamically optimized and adjusted to obtain optimized support sample features and query sample features. The weight allocation module is used to calculate the similarity between the support sample features and the query sample features based on the optimized support sample features and query sample features, reallocate the sample weights based on the similarity between the support sample features and the query sample features, and obtain the reallocated support sample features and query sample features based on the sample weights. The action recognition module is used to calculate the frame-level distance between the support sample features and the query sample features based on the reassigned support sample features and query sample features. It uses the Dynamic Time Warping (DWT) algorithm to dynamically plan the frame-level distance between the support sample features and the query sample features, and obtains the total distance between the support set and the query set as the skeleton action recognition result.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs the few-sample skeleton action recognition method based on a dual attention focus network as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is loaded and executed by the processor, it employs the few-sample skeleton action recognition method based on a dual attention focus network as described in any one of claims 1 to 7.

Images