Large language model skeleton action recognition method, system and electronic device

Abstract

This invention relates to a method, system, and electronic device for skeleton action recognition using a large language model (LLM). Addressing three major problems in current skeleton action recognition methods when incorporating a large language model (LLM): a disconnect between semantic guidance and spatial topology learning, a lack of hierarchical semantic support in temporal modeling, and insufficient generalization ability of traditional classification paradigms, this invention proposes a novel skeleton action recognition method (LLMA-GCN) based on LLM enhancement. First, this method proposes a hybrid graph topology learning strategy based on LLM, deeply fusing prior knowledge of the physical skeleton structure with action semantic information extracted by LLM to guide spatial modeling. Second, it combines the hybrid graph topology learning strategy with a hierarchical visual sequence encoder, achieving multi-scale spatiotemporal feature extraction under semantic guidance through designed LLM refinement blocks. Finally, it constructs an LLM-driven action-text prototype-guided decision learning mechanism to achieve joint learning of semantically guided visual-text alignment and classification decisions, thereby improving the overall performance of the model.

Description

Technical Field

[0001] This invention belongs to the field of human motion recognition technology, and relates to skeleton motion, especially a method, system and electronic device for skeleton motion recognition of large language models. Background Technology

[0002] Human action recognition, as one of the core research directions in computer vision, has extremely important application value in fields such as intelligent surveillance and human-computer interaction. Among them, action recognition methods based on skeleton data have gradually become a research hotspot due to their strong robustness to factors such as background changes and lighting conditions. In order to further expand the semantic information of skeleton actions, researchers have introduced Large Language Models (LLMs). However, the text features extracted by LLMs are essentially macroscopic, one-dimensional, sequential, and static semantic representations, while skeleton actions are microscopic, dynamic spatiotemporal graph structures. This makes it difficult to align the high-level semantic information generated by LLMs with the underlying physical topology of the skeleton at a fine-grained level, resulting in the following limitations:

[0003] 1. Existing methods rely on fixed physical topology to construct graph structures for spatial modeling. The semantic information introduced based on LLM is only used as an auxiliary feature branch and is not directly aligned with the visual backbone. It cannot integrate semantic information to guide the dynamic learning of graph structures, resulting in a disconnect between semantic guidance and graph topology learning in the spatial dimension.

[0004] 2. Methods for time modeling through multi-scale action recognition typically use it as a general module in the network, ignoring the semantic differences in spatial features at different network depths, resulting in a lack of effective hierarchical semantic guidance in the time modeling process.

[0005] 3. Existing methods rely solely on fully connected layers or semantic information for classification, failing to maximize the advantages of both and limiting the model's generalization ability. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system and electronic device for large language model skeleton action recognition with strong generalization ability and accurate recognition.

[0007] To solve the above problems, the technical solution of the present invention is as follows:

[0008] A method for skeleton action recognition of a large language model includes the following steps:

[0009] The semantic adjacency matrix of joints and the semantic vector of actions are obtained through a large language model, and the physical skeleton adjacency matrix is ​​extracted through skeleton action data.

[0010] A hybrid adjacency matrix for skeleton actions is constructed by combining the semantic adjacency matrix of joints and the physical skeleton adjacency matrix, which is used to enhance the features of skeleton actions.

[0011] Construct a hierarchical visual sequence encoder to obtain visual features;

[0012] Visual features are aligned with visual semantic features based on action semantic vectors, and the recognition information and prediction results of skeleton actions are output.

[0013] In a further embodiment, a hybrid adjacency matrix for skeleton actions is constructed by extracting the physical skeleton adjacency matrix from the joint semantic adjacency matrix and the skeleton action data. This matrix is ​​used to enhance the features of the skeleton actions, specifically including:

[0014] A parameter matrix is ​​constructed by considering the importance of different physical connections in skeletal motion. The parameter matrix is ​​then multiplied element-wise with the physical skeleton adjacency matrix to obtain an adaptive adjacency matrix.

[0015] A hybrid adjacency matrix is ​​obtained based on the adaptive adjacency matrix and the keypoint semantic adjacency matrix.

[0016] In a further embodiment, a hierarchical visual sequence encoder is constructed to obtain visual features, specifically including:

[0017] The hierarchical visual sequence encoder consists of three stages, each of which comprises three stacked refinement blocks, each based on a large language model.

[0018] In a further embodiment, each refining block includes three parallel graph convolution branches and three parallel dynamic convolution branches.

[0019] The input features of the refined block are passed through three graph convolution branches and then added element by element to obtain the feature aggregation;

[0020] Feature aggregation establishes residual connections with input features to obtain intermediate features;

[0021] The intermediate features are input into three dynamic convolutional branches respectively, and then concatenated to obtain multi-scale temporal features;

[0022] The multi-scale temporal features are connected with the input features and intermediate features through a double residual concatenation to obtain the output features.

[0023] In a further embodiment, the three dynamic convolutional branches employ 3×1, 5×1, and 7×1 convolutional kernels, respectively.

[0024] In a further embodiment, a hierarchical visual sequence encoder is used to extract joint flow data, joint motion flow data, skeletal flow data, and skeletal motion flow data of skeletal movements to obtain visual features.

[0025] In a further embodiment, visual-semantic feature alignment processing is performed on visual features based on action semantic vectors to output recognition information and prediction results of skeleton actions, specifically including:

[0026] The visual features are input into the fully connected classifier and the visual feature projector, respectively.

[0027] The classification logic value is obtained through a fully connected classifier, and the visual feature vector is obtained through a visual feature projector.

[0028] The action semantic vector is normalized and embedded to obtain the action text prototype. The action text prototype is then projected onto the visual-semantic space shared with the visual feature vector through a text feature projector, and the action text feature vector is output.

[0029] Calculate the cosine similarity between the visual feature vector and the action text feature vector to obtain the text alignment logical value.

[0030] In a further embodiment, visual-semantic feature alignment processing is performed on visual features based on action semantic vectors to output skeleton action recognition information and prediction results, and the method further includes:

[0031] Construct the classification cross-entropy loss using classification logistic values;

[0032] Construct the text alignment cross-entropy loss using text alignment logical values;

[0033] The classification cross-entropy loss and the text alignment cross-entropy loss are weighted to obtain a joint loss, which completes the visual-semantic feature alignment process.

[0034] A large language model skeleton action recognition system, which executes a large language model skeleton action recognition method, includes:

[0035] The hybrid graph topology learning module is used to construct a hybrid adjacency matrix for skeleton actions;

[0036] The encoding module is used to build a hierarchical visual sequence encoder to obtain visual features;

[0037] The visual-semantic alignment module is used to perform visual-semantic feature alignment processing on visual features.

[0038] An electronic device, comprising:

[0039] At least one processor;

[0040] Memory that is communicatively connected to at least one processor;

[0041] The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to:

[0042] The semantic adjacency matrix of joints and the semantic vector of actions are obtained through a large language model, and the physical skeleton adjacency matrix is ​​extracted through skeleton action data.

[0043] A hybrid adjacency matrix for skeleton actions is constructed by using a joint semantic adjacency matrix and a physical skeleton adjacency matrix, which is used to enhance the features of the skeleton actions.

[0044] Construct a hierarchical visual sequence encoder to obtain visual features;

[0045] Visual features are aligned with visual semantic features based on action semantic vectors, and the recognition information and prediction results of skeleton actions are output.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] 1. This recognition method firstly integrates physical skeleton priors with action semantic information through an LLM-guided hybrid graph topology learning strategy; secondly, it combines a hierarchical visual sequence encoder with LLM refinement blocks to achieve multi-scale spatiotemporal feature extraction under semantic guidance; finally, it constructs an LLM-driven action-text prototype-guided decision learning mechanism to complete the joint learning of semantic-guided visual-text alignment and classification decisions. It systematically integrates LLM semantic knowledge from three levels: spatial modeling, temporal modeling, and decision classification, effectively addressing the shortcomings of existing methods. Extensive comparative and ablation experiments on multiple public datasets, including NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD, have validated its effectiveness and advancement, significantly improving the accuracy and generalization ability of skeleton action recognition.

[0048] 2. This recognition method constructs a semantic adjacency matrix for joints and adds it to the physical skeleton adjacency matrix to obtain a hybrid adjacency matrix. This enables the network to simultaneously perceive the prior physical connections between joints and the semantic functional associations of actions extracted by LLM. It effectively overcomes the shortcomings of traditional GCN methods that rely solely on fixed physical topology and ignore the semantic correlation of actions. This allows spatial modeling in the spatiotemporal graph feature learning process to obtain direct guidance from semantic information, thereby extracting more discriminative action feature representations and significantly improving the performance of semantically enhanced action recognition.

[0049] 3. This recognition method combines a hybrid graph topology learning strategy with a hierarchical visual sequence encoder. Using skeleton flow data as input, it employs parallel multi-scale spatial graph convolution and hierarchical temporal feature extraction guided by a hybrid adjacency matrix. Furthermore, by leveraging the hierarchical stacking of multi-stage LRB refinement blocks, it simultaneously achieves semantically information-guided multi-scale spatiotemporal feature learning in both spatial and temporal dimensions. This design effectively addresses the problem of semantic information loss in traditional temporal modeling, enabling the model to obtain semantically perceptive spatiotemporal representations at different temporal granularities, thus enhancing the model's dynamic understanding of complex action sequences.

[0050] 4. This recognition method aligns the features output by the visual sequence encoder with the action text prototypes generated based on LLM within a shared multimodal semantic space. A joint loss function is designed to achieve collaborative optimization of the visual and textual semantic branches. This mechanism breaks through the traditional action category decision boundary learning mode based on pure visual features, transforming it into a joint learning of visual-semantic alignment and classification decisions guided by action text prototypes. This significantly improves the model's generalization ability to unseen samples and cross-scene actions, further enhancing the overall performance of the model. Attached Figure Description

[0051] Figure 1 A flowchart of a large language model skeleton action recognition method;

[0052] Figure 2 This is a structural diagram of a large language model skeleton action recognition method.

[0053] Figure 3 A hierarchical visual sequence encoder structure diagram for a large language model skeleton action recognition method;

[0054] Figure 4 This is an LRB refinement block structure diagram for a large language model skeleton action recognition method.

[0055] Figure 5(a) is a visualization of the 1-hop physical skeleton adjacency matrix of a large language model skeleton action recognition method;

[0056] Figure 5(b) is a visualization of the 2-hop physical skeleton adjacency matrix of a large language model skeleton action recognition method;

[0057] Figure 5(c) is a visualization of the physical skeleton adjacency matrix of a large language model skeleton action recognition method;

[0058] Figure 5(d) is a visualization of the semantic adjacency matrix of a large language model skeleton action recognition method;

[0059] Figure 5(e) is a visualization of the initial weighted hybrid adjacency matrix of a large language model skeleton action recognition method;

[0060] Figure 5(f) is a visualization of the initial weighted hybrid adjacency matrix of a large language model skeleton action recognition method;

[0061] Figure 5(g) is a visualization of the initial weighted hybrid adjacency matrix of a large language model skeleton action recognition method. Detailed Implementation

[0062] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0063] Human motion recognition, as one of the core research directions in computer vision, has extremely important application value in fields such as intelligent surveillance and human-computer interaction. Based on the different types of input data, existing motion recognition methods can be divided into RGB video-based methods, skeleton data-based methods, and methods based on multi-source data fusion. In recent years, skeleton data-based motion recognition methods have gradually become a research hotspot due to their strong robustness to background changes and lighting conditions. Compared with RGB video-based methods, skeleton data can more directly describe the motion characteristics of the human body, and its computational complexity is relatively low, making it applicable to more real-world scenarios.

[0064] Research methods for skeleton motion recognition have undergone a significant evolution from early handcrafted features to deep learning. Early research mainly relied on using handcrafted features (3D coordinates of joints, trajectories over time series, etc.) to describe the spatiotemporal dynamics of motion. While existing methods perform well in specific scenarios, their ability to represent skeleton data is limited, and their poor generalization makes it difficult to recognize multi-class motions in complex scenes.

[0065] To further expand the semantic information of skeletal movements and compensate for the semantic deficiencies in the representation of pure physical coordinate data, researchers attempted to enhance feature representation using a Large Language Model (LLM). Compared to traditional semantic extraction models (such as Word2Vec), LLM possesses rich open-world prior knowledge and powerful reasoning capabilities. This allows it to overcome the limitation of contextual information in traditional word embedding models based on short text containing only action labels or joint names, thereby deeply exploring the functional associations and interaction logic of human body parts in specific movements. This top-down common-sense prior provides an effective semantic supplement to highly abstract 3D skeletal data lacking visual details, thus shifting the model's action recognition paradigm from purely data-driven to knowledge-driven.

[0066] However, despite LLM's significant advantages in supplementing high-level semantic information, it still faces the challenge of cross-modal fusion in practical applications of skeleton action recognition. The text features extracted by LLM are essentially macroscopic, one-dimensional, sequential, static semantic representations, while skeleton actions are microscopic, dynamic spatiotemporal graph structures. This difference in granularity and structure makes it difficult to fine-grained align the high-level semantic information generated by LLM with the underlying physical topology of the skeleton. Due to this limitation, existing methods face the following main constraints in achieving visual-semantic alignment and deep fusion. First, semantic guidance is disconnected from graph topology learning in the spatial dimension. Existing methods rely on fixed physical topologies to construct graph structures for spatial modeling. The semantic information introduced by LLM serves only as auxiliary feature branches and is not directly aligned with the visual backbone, failing to integrate semantic information to guide the dynamic learning of the graph structure. Second, effective hierarchical semantic guidance is lacking in temporal modeling. Although some studies have explored multi-scale action recognition temporal modeling, it is usually used as a general module in the network, ignoring the semantic differences of spatial features at different network depths, resulting in a lack of semantic guidance in the temporal modeling process. Finally, existing methods rely solely on fully connected layers or semantic information for classification, failing to maximize the advantages of both and limiting the model's generalization ability.

[0067] Example 1:

[0068] A method for skeleton action recognition of a large language model, such as Figure 1 As shown in Figure 5(g), the LLMA-GCN framework is constructed, which is a graph convolutional framework based on large language model enhancement. By deeply utilizing the semantic prior knowledge derived from LLM and the inherent physical topological features of 3D skeleton data, the accuracy and robustness of the model in multi-class action recognition in complex scenes are comprehensively improved. Its overall algorithm architecture is as follows: Figure 2As shown, LLMA-GCN uses bimodal data of action visual skeletons and text as training input. Previous research has shown that fine-tuning LLM using few-sample data of a specific modality is prone to catastrophic forgetting, impairing its general open-world knowledge, while reasoning patterns based on frozen parameters can effectively stimulate its inherent semantic priors. Therefore, for text modal information, the LLMA-GCN framework utilizes designed joint cue words and action cue words to guide the large language model to offline extract and pre-compute two core components before training: a joint semantic adjacency matrix and an action semantic vector library. The joint semantic adjacency matrix is ​​used to capture the functional and holistic relationships between joints, while the action semantic vector library contains semantic information of different types of actions. Specifically, the joint semantic adjacency matrix is ​​fused with the physical skeleton adjacency matrix based on the action skeleton to form a hybrid adjacency matrix, which is used to guide subsequent hierarchical visual sequence encoding; the action semantic vector library is used to guide visual-semantic feature fusion and classification alignment.

[0069] To overcome the representational bottleneck of single skeleton data for visual modal information, four classic skeleton flow data types are obtained through preprocessing of the original skeleton flow data: joint flow, joint motion flow, skeleton flow, and skeleton motion flow. A hierarchical visual sequence encoder based on LLM refinement blocks (LRBs) is then used to extract features from these four skeleton flow data types in parallel. The hierarchical visual sequence encoder consists of three stages of hierarchical extraction of spatiotemporal features, with each stage composed of three cascaded stacked LRBs. Using the skeleton data as input, feature aggregation is achieved through spatial graph convolution guided by a hybrid adjacency matrix, realizing multi-stage, progressive spatiotemporal feature enhancement of the skeleton features. This architecture enables the model to adaptively learn complex multi-scale spatiotemporal action sequence features enhanced by joint semantic information.

[0070] Finally, in the multimodal feature semantic alignment and action classification stage, several high-order visual features extracted by the hierarchical visual sequence encoder are global average pooled and then input into a fully connected classifier and a visual feature projector constructed from a multilayer perceptron, respectively. The former outputs the classification logits. FC The latter obtains the visual feature vector f in the visual-semantic multimodal space through mapping. v To achieve semantically guided feature alignment and classification learning, a text feature projector constructed from a multilayer perceptron is used to project the action text prototype onto the corresponding f. v 'Shared visual-semantic space, outputting action text feature vector f' t Then, by calculating f... v 'and f t The cosine similarity between the two is used to obtain the text alignment logical value Logits. text Logits FC and Logits textThey are used to construct the classification cross-entropy loss L. cls Text alignment and cross-entropy loss L text And the weighted joint loss L is obtained. total The model in L total Achieving visual-semantic feature alignment under constraints. That is, the model utilizes L... text To assist in supervising the signal, the visual feature main classification branch ultimately outputs the action prediction result with the highest probability.

[0071] Specifically, it includes the following steps:

[0072] S101. Obtain the semantic adjacency matrix of key points through a large language model, and extract the physical skeleton adjacency matrix through skeleton motion data:

[0073] Traditional graph convolution-based skeleton action recognition networks often rely solely on sparse physical connections or use purely data-driven adaptive graphs when extracting spatial features of actions. The former has a limited receptive field, while the latter lacks stability and is prone to overfitting. To address this issue, this application proposes a hybrid graph topology learning strategy based on LLM, including:

[0074] 1. Joint Semantic Adjacency Matrix: Joint cue words are obtained through joint names. These cue words are then input into an LLM (Local Link Model), which generates the joint semantic adjacency matrix. Specifically: In the formula This is the semantic adjacency matrix of the key points; It is the set of real numbers; The number of joints is given. To enable the graph structure to perceive the functional relationships between joints, such as the coordination between the left hand and right foot when walking, semantic knowledge of LLM is introduced. Furthermore, this embodiment uses the NTU RGB+D and PKU-MMD public datasets for testing: For 25 joints in the NTU RGB+D and PKU-MMD datasets, a unified joint cue word template was designed: "human body joint: {name}"; the joint name is processed by the joint cue word template and then input into LLM, and then trained offline by the lightweight sentence embedding model all-MiniLM-L6-v2 to obtain a text embedding representation E containing the functional relationships between all relevant nodes. sent Then, by calculating the cosine similarity, a semantic adjacency matrix of the key points is constructed, as shown in formula (1):

[0075] (1)

[0076] In the formula, Let E be the semantic adjacency matrix of the joints; u, v are the indices of the joints, u, v ∈ {1,…,V}; sentThis is a text embedding representation; `name` represents the corresponding name of each of the 25 joints. The joint semantic adjacency matrix is ​​a dense matrix based on continuous values, assigning higher connection weights to anatomically non-adjacent but symmetrical joints. This global structural prior knowledge based on the inherent semantic similarity of human joints does not depend on specific training data and possesses good reliability and cross-dataset generalization ability.

[0077] 2. Physical skeleton adjacency matrix (or simply physical adjacency matrix): Extracted based on the input raw skeleton data, specifically: In the formula The physical adjacency matrix; The number of subsets of the physical adjacency matrix is ​​given in this embodiment. The value is set to 3, which means it contains skeleton information in three dimensions; R is the set of real numbers; This represents the number of joints. The physical adjacency matrix is ​​constructed based on the physical structure of the human skeleton, with three dimensions of skeleton information: A phy0 Represents the node itself, used to preserve the node's own feature information; A phy1 Representing first-order physical adjacency, i.e., directly connected joints, used to capture local fine-grained physical dependencies; A phy2 Representing second-order physical adjacency, i.e., joints connected by a single joint, used to establish long-distance global dependencies. Unlike dense semantic adjacency matrices, A... phy It is sparse and reflects the physically immutable anatomical topological relationships between joints, such as the wrist being connected only to the elbow. The physical adjacency matrix provides prior knowledge of human anatomy for the skeletal model. For example, in the "waving" motion, the physical connections between the wrist, elbow, and shoulder are key to understanding the movement, and these connections are precisely encoded in A. phy middle.

[0078] S103. Construct a hybrid adjacency matrix for skeleton actions using the semantic adjacency matrix of joints and the physical skeleton adjacency matrix, which is used to enhance the features of skeleton actions:

[0079] Construct a learnable parameter matrix: In the formula It is a learnable parameter matrix; The number of subsets of the physical adjacency matrix is ​​given in this embodiment. The value is set to 3, which means it contains skeleton information in three dimensions; R is the set of real numbers; This represents the number of joints. With A phyFor the same size, it can be optimized end-to-end along with the network. The parameter matrix does not directly represent connections, but rather quantifies the importance of different physical connections in a learnable way. Larger values ​​in the parameter matrix mean increased information flow on that connection, while smaller values ​​mean decreased information flow. By using A... phy Element-wise multiplication with the learnable parameter matrix yields the adaptive adjacency matrix A. adapt As shown in formula (2):

[0080] (2)

[0081] In the formula, An adaptive adjacency matrix; The physical adjacency matrix; It is a learnable parameter matrix; This is an element-wise product.

[0082] Finally, A adapt and A sem By weighting the data using learnable parameters α, we obtain the final hybrid adjacency matrix A used to guide skeleton feature learning. hyb Its k-th subgraph (k=1,…,K) is shown in formula (3):

[0083] (3)

[0084] In the formula, Let be the mixed adjacency matrix of the k-th subgraph; For the Sigmoid function; These are learnable parameters; Let be the adaptive adjacency matrix of the k-th subgraph; This is the k-th subgraph; This is the semantic adjacency matrix of the joints.

[0085] The skeleton action hybrid adjacency matrix is ​​used by the model to extract skeleton action features enhanced by the semantic information of the joints, so that the semantic information of the joints can directly participate in the subsequent spatial graph convolution and temporal feature extraction, thereby realizing dynamic learning of hybrid graph topological spatiotemporal features based on LLM.

[0086] S105. Construct a hierarchical visual sequence encoder to obtain visual features:

[0087] 1. Visual sequence encoder:

[0088] Visual sequence encoders can achieve progressive feature enhancement, that is, they learn the spatiotemporal feature representation of skeleton sequences in depth through a staged hierarchical architecture. For example... Figure 3As shown, the visual sequence encoder consists of three cascaded stages from top to bottom, each stage being constructed by stacking three LRB refinement blocks. The hierarchical "progressive" feature extraction of the visual sequence encoder is manifested in the following two aspects:

[0089] Progressive feature dimension: From stage one to stage three, the number of channels in the feature map is gradually increased. This enables the network to learn hierarchical representations of action units, from low-order, simple action units to high-order, complex semantic-level action units, within an ever-expanding feature space.

[0090] Progressive receptive field: Stages two and three downsample the feature maps by using convolutions with a stride of 2 in the temporal dimension. This progressively reduces the temporal resolution of the feature maps, allowing subsequent convolutional layers to cover longer time windows—that is, the temporal receptive field is progressively expanded. This enables the model to progressively transition from focusing on local, fast action details to understanding global, long-term action categories.

[0091] The core building block of each stage is the LRB module, which processes the input feature X. in Skip connections and multiple fusions are used to extract spatiotemporal features at each level, such as... Figure 4 As shown. Input feature X in Feature aggregation is achieved through element-wise addition via three parallel graph convolution branches, guided by a hybrid graph adjacency matrix. This aggregation is then combined with the input feature X. in Establish residual connections to obtain intermediate features Y mid Then, Y mid The inputs are fed into three dynamic convolutional branches, each independently extracting time-series features at its corresponding scale. Finally, the output features of all dynamic branches are concatenated along the channel dimension to obtain the multi-scale time-series feature Y. multi Then, combined with the input feature X in and intermediate features Y mid Perform a dual residual connection to obtain the output feature Y of the LRB module. out This process achieves semantic enhancement of multi-layered spatiotemporal features of skeleton visual sequence information based on a hybrid graph topology learning strategy.

[0092] 2. Hierarchical temporal feature extraction module:

[0093] In action recognition tasks, temporal dynamics exhibit high diversity. To match the spatial dimension features extracted based on hybrid graph topology, a hierarchical temporal extraction module based on multi-scale convolution is introduced into the network's temporal branch. Within each LRB, the hierarchical temporal feature module is used collaboratively to learn the temporal features of different actions guided by semantic information, such as... Figure 4As shown, feature learning at different time scales is decoupled by using three parallel convolutional branches, each responsible for a specific time scale. Specifically, this includes:

[0094] Local short-term dynamic branch: Employs a smaller 3×1 convolution kernel to focus on capturing fine-grained, rapidly changing motion features Y3;

[0095] Mid-range motion transition branch: Uses a medium-sized 5×1 convolution kernel, suitable for modeling motion features Y5 of moderate duration that constitute the core part of most motions;

[0096] Global motion rhythm branch: Using a larger 7×1 convolution kernel, it can cover a longer time window to understand continuous motion features with macroscopic regularity Y7;

[0097] After acquiring features at the three different temporal granularities, the hierarchical temporal feature extraction module concatenates them along the channel dimension to obtain the multi-scale temporal feature Y. multi As shown in formula (4):

[0098] (4)

[0099] In the formula, It represents multi-scale temporal features; For feature splicing and fusion operations; Characteristics of motion; Characteristics of motion; It refers to action characteristics.

[0100] Subsequently, batch normalization (BN) is performed sequentially on the concatenated features to stabilize the deep feature distribution and accelerate convergence, while the Dropout mechanism is used to avoid overfitting in multi-scale feature learning. Simultaneously, to overcome the gradient vanishing problem easily caused by the superposition of multi-layer graph convolution and temporal convolution in deep spatiotemporal networks, and to maximize the utilization of original feature information, a classic dual residual connection mechanism is used in the LRB refinement block, such as... Figure 4 As shown. Therefore, the output feature Y of the LRB module out As shown in formula (5):

[0101] (5)

[0102] In the formula, For output features; It uses the Dropout mechanism; It is a linear rectified function; For batch normalization; It represents multi-scale temporal features; For internal time residuals, corresponding Figure 4 The skip connection above the temporal feature extraction module can incorporate the features output by the graph convolution module. It is directly added to the normalized result of the multi-scale temporal stitching feature to preserve the inter-frame temporal structure of the baseline; For intermediate features; R LRB (X in ) represents the external block-level residual, corresponding to Figure 4 The bottommost long skip connection spans the entire graph convolution and temporal convolution modules, injecting the most original skeleton sequence features into the final output; X in This represents the features of the input LRB.

[0103] When the feature space dimension increases, i.e., when the network enters a deeper stage requiring temporal downsampling or a change in the number of channels, R... tcn (·) and R LRB Alignment is achieved using 1×1 convolutions; otherwise, identity mapping is used. This dual residual design enables the model to balance the spatiotemporal details of local actions with the semantic features of macroscopic actions. Simultaneously, it provides robust action skeleton network topology features for the collaborative optimization of LLM-based hybrid graph topology learning and hierarchical visual sequence encoding.

[0104] Therefore, the combined use of a hybrid graph topology learning strategy based on LLM and a hierarchical visual sequence encoder synergistically enhances the representation of skeleton motion features from both spatial and temporal dimensions. Spatially, by introducing a joint semantic adjacency matrix and fusing it with the physical adjacency matrix to obtain a hybrid adjacency matrix, the model, when performing spatial convolution operations, essentially transmits information on a graph with richer prior knowledge. For key joints sensitive to specific actions, such as the left and right hands in a "clapping" motion, their information can directly interact and fuse features in a single-layer convolution, thereby improving the model's ability to extract discriminative spatial motion posture features. Temporally, the features output by the graph convolution enhanced with joint semantics include a global collaborative representation based on LLM semantic similarity aggregation, enabling the hierarchical temporal modeling module to more effectively capture the temporal dynamics during action execution and exhibit stronger robustness to time shifts, occlusion, or irregular sampling rates. This allows for better joint learning of the evolution of whole-body functional movements over time, rather than being limited to learning the motion features of single joints or local limbs, giving the model stronger generalization ability at the spatiotemporal joint level.

[0105] S107. Based on the action semantic vector, perform visual-semantic feature alignment processing on the visual features, and output the recognition information and prediction results of the skeleton action:

[0106] To enhance the feature representation richness of traditional fully connected layer classifiers in action recognition tasks, thereby improving the model's generalization ability, this application also includes an LLM-driven action-text prototype-guided decision learning mechanism. This mechanism breaks through the traditional single paradigm that relies solely on visual features to learn decision boundaries. By constructing a dual-branch joint learning framework for visual classification and text alignment, it utilizes high-level semantic knowledge of actions to calibrate the underlying visual motion feature classification, thus achieving deep alignment of visual and semantic features and classification decision learning. The overall process is as follows: Figure 2 As shown on the right.

[0107] For each action category in the dataset, a set of semantic vector representations, i.e., action text prototypes, are constructed according to different dimensions. First, for the name of the i-th action category, three complementary action cue word templates are designed to promote LLM's understanding of actions from three dimensions: descriptive, observable, and declarative. The specific configuration of action text prototype cue words is shown in Table 1.

[0108] Table 1

[0109]

[0110] Multiple action semantic vectors are obtained by text encoding several cue words using LLM, and then an aggregated action text prototype is calculated by averaging the semantic vectors for that category. Specifically: In the formula R is the prototype of the action text; R is the set of real numbers; This represents the hidden layer dimension of the LLM. After L2 normalized embedding, all categories of action text prototypes collectively form a fixed action text prototype library. In the formula R is a prototype library for action text; R is the set of real numbers. The hidden layer dimension of the LLM; This represents the total number of action categories. The feature alignment and joint optimization process is as follows:

[0111] 1. Visual Spatiotemporal Feature Aggregation: During training and inference, the input skeleton data is processed by a hierarchical visual sequence encoder to extract features f. v It aggregates features in the spatiotemporal dimension through global average pooling, which is a dense visual spatiotemporal representation at the video level.

[0112] 2. Multimodal Unified Space Mapping: The action text prototype library P can be regarded as a set of semantic information points in the text feature space. After passing through the text feature projector, the action text feature vector f is obtained. t Simultaneously, a visual feature projector is used to project dense visual features f v Mapping to f tIn a shared multimodal semantic space of the same dimension, the visual feature vector f is obtained. v '.

[0113] 3. Cross-modal similarity calculation: Within a unified high-dimensional feature space, the visual feature vector f is calculated. v 'and action text feature vector f t The cosine similarity of ' is used to obtain the cross-entropy loss L used to construct the text alignment. text Logits (text alignment logical value) text As shown in formula (6):

[0114] (6)

[0115] In the formula, τ is the text alignment logical value; τ is a learnable temperature coefficient used for automatic adjustment during training. The sharpness of the distribution constitutes the text alignment branch output of the network; f v ' f is a visual feature vector; t 'T This is the transpose of the action text feature vector.

[0116] 4. Perform dual-branch collaborative optimization: such as Figure 2 As shown, visual feature f v Logits obtained after a fully connected classifier FC Used to construct the classification cross-entropy loss L cls The action prediction result is obtained by passing argmax. During the network training phase, the classification cross-entropy loss L is used. cls Cross-entropy loss L of the text-aligned branch text Combined, construct the overall joint loss L of the network total That is, by using the high-level semantic features of the text as a regularization term, visual features in the multimodal space are clustered towards the textual semantic information points corresponding to their correct categories, while moving away from irrelevant action categories, thereby achieving dual-branch collaborative optimization of visual and textual semantics. The overall joint loss of the network is L. total As shown in formula (7):

[0117] (7)

[0118] In the formula, The overall network suffers a combined loss; For classification cross-entropy loss; For text alignment cross-entropy loss; This is the balance coefficient.

[0119] The present invention also provides a computer device, including a processor component comprising at least one processor; a storage component comprising a memory; program instructions comprising a computer program stored in the memory; the processor component being configured to read and execute the program instructions in the storage unit to implement steps S101 to S107.

[0120] The present invention provides a computer program product including computer-executable instructions, which, when executed on a processor of an electronic device, enable the electronic device to perform steps S101 to S107.

[0121] This invention also provides a computer-readable storage medium storing computer program instructions. The storage medium includes non-volatile memory (e.g., solid-state drives, flash memory) or temporary storage media (e.g., memory cache). The computer-readable storage medium stores an independent product, which implements steps S101 to S107 in the form of software functional units. Essentially, the part of the technical solution of this invention that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute steps S101 to S107. The aforementioned storage medium includes, but is not limited to, semiconductor memories (e.g., ROM, RAM, Flash), magnetic storage media (e.g., hard disks, magnetic tapes), optical storage media (e.g., CDs, DVDs), and other physical storage devices capable of carrying program code (e.g., USB flash drives, portable hard drives, etc.).

[0122] This invention uses three publicly available datasets: NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD to conduct model performance evaluation experiments.

[0123] 1. Comparative experiment:

[0124] To verify the advancement of the proposed LLMA-GCN model in skeleton action recognition, sixteen models were selected for comparative experimental analysis. These include: seven GCN-based improved skeleton action recognition models: ST-GCN, DS-GCN, DSDC-GCN, STA-GCN-Transformer, BlockGCN, SMS-GCN, and TSGCNeXt; six LLM-enhanced skeleton action recognition models: GAP, MICA, CEPCLR, HS-Rep, LLM-AR, and KEHCN; and three multimodal action recognition models based on RGB video data: MMNet, MMINet, and TBCNet. The comparative experimental results are shown in Table 2, representing the results (%) on three publicly available datasets. To maintain fairness and consistency in the comparison, experimental results from the original literature are directly cited. It should be noted that the datasets used in the original literature for some methods are not entirely the same as those used in this application; a "-" indicates that the original literature does not include this data.

[0125] Table 2

[0126]

[0127] As shown in Table 2, compared with the other thirteen skeleton action recognition methods, the LLMA-GCN model achieved the best recognition results on all three public datasets. Even compared with the three multimodal action recognition methods based on RGB video data, the LLMA-GCN model still showed competitive performance advantages, which verified the effectiveness of the LLMA-GCN model.

[0128] (1) On the NTU RGB+D 60 dataset, which has fewer action categories, the LLMA-GCN model achieves the best experimental results on the X-sub benchmark. The experimental results of different models on the NTU RGB+D 120 dataset are relatively poor. This is because the dataset contains more action categories, more subjects, and more complex experimental scenarios, which places higher demands on the model's generalization ability. The LLMA-GCN model still achieves a high level of recognition accuracy under the X-sub and X-set scene divisions of this dataset. Comparing the experimental results of different models on this dataset, it can be found that using RGB video of the action as input can effectively improve the model's recognition performance. At the same time, LLMA-GCN uses only skeleton data as input, and its action recognition accuracy exceeds that of the other two models that use RGB video (MMINet, TBCNet). This shows that the LLMA-GCN model can effectively improve the generalization and robustness of skeleton action recognition in complex scenarios.

[0129] (2) Performance analysis on the PKU-MMD I dataset

[0130] Compared to the NTU RGB+D 120 dataset, PKU-MMDⅠ contains fewer action categories and fewer subjects, yet different models achieve higher recognition accuracy on this dataset. LLMA-GCN achieves the highest recognition accuracy of 98.3% on the PKU-MMDⅠ X-sub benchmark and 97.4% on the X-view benchmark. Similarly, the experimental results on this dataset also show that using RGB video as input improves the model's understanding of actions; the LLMA-GCN model, through its designed LLM-driven semantic enhancement method, effectively improves skeleton action recognition performance, thus approaching or exceeding the performance of multimodal action recognition models based on RGB video.

[0131] Meanwhile, compared to seven other graph convolution-based skeleton action recognition models, the LLMA-GCN model shows a significant overall performance improvement across the three datasets. This is because traditional GCN-based skeleton action recognition models primarily rely on physical adjacency matrices or data-driven dynamic graph structures, which are limited by fixed physical connections. Therefore, in complex interaction scenarios, the LLM-driven semantic enhancement method based on a hybrid graph topology proposed by the LLMA-GCN model can more accurately model the spatiotemporal features of actions, thereby effectively improving the performance of skeleton-based action recognition.

[0132] 2. Ablation experiment:

[0133] To verify the effectiveness of the hierarchical visual sequence encoder (M), the LLM-based hybrid graph topology learning strategy (S), and the LLM-driven action-text prototype-guided decision learning mechanism (T) designed in the LLMA-GCN model, ablation experiments were conducted on the X-sub benchmark of the PKU-MMD I dataset. A spatiotemporal graph convolutional network (B) with physical adjacency matrix and standard temporal convolution as the backbone structure and four-modal skeleton data as visual input was used as the baseline model. These components were progressively added to the B model. The experimental results are shown in Table 3, representing the ablation experiment results (%) of the LLMA-GCN framework.

[0134] Table 3

[0135]

[0136] It can be seen that the introduction of each component brought varying degrees of performance gains, validating the rationality of the proposed framework. Specifically, ① compared with the baseline model, the introduction of M, S, and T modules individually all resulted in performance improvements. Among them, the S module brought the most significant gain (the recognition accuracy increased from 90.8% to 96.8%, an improvement of 6%), which verifies that using LLM prior knowledge to construct semantic-level physical joint connections can effectively overcome the limitations of traditional physical skeleton connections and help the model further learn action features. At the same time, the M and T modules improved the action recognition accuracy by 3.1% and 3.5%, respectively. ② In the experimental combination using two components, B+S+T achieved a high accuracy of 97.8%, indicating a good complementary effect between the action text prototype and the semantic topology, which jointly enhances the semantic discriminative power of the features. ③ Comparing the experimental results of B+S and B+M+S, it was found that introducing hierarchical visual sequence encoding on top of the LLM-based hybrid graph topology learning strategy resulted in only a small improvement in model performance; however, the LLMA-GCN model (i.e., the last row of Table 2, B+M+S+T) achieved the best synergistic effect (98.3%), surpassing all single-component and dual-component combinations. This indicates that, under the specification and guidance of the action text prototype (T), the model can simultaneously utilize hierarchical multi-scale temporal information (M) and deep semantic information (S) to extract a more robust and unified spatiotemporal feature representation of actions.

[0137] 3. Detailed ablation analysis of different modules:

[0138] Experiment on weight fusion in mixed adjacency matrix:

[0139] In the construction of the hybrid adjacency matrix, a learnable fusion weight parameter α is introduced to dynamically balance the contributions of the semantic and physical adjacency matrices in feature aggregation. During training, α is automatically updated based on the gradient signal from backpropagation, enabling the model to adaptively learn the optimal semantic-physical information fusion ratio. To systematically verify the impact of different semantic-physical fusion ratios and the use of parameter α on action recognition performance, three different semantic adjacency matrix proportions (i.e., three initial values ​​of α) were set in the experiment. Experiments were conducted under the conditions of a fixed fusion ratio (i.e., a fixed value for α) and automatic learning of parameter α. The experimental results are shown in Table 4, which presents the experimental analysis results (%) of the fusion weight in the hybrid adjacency matrix.

[0140] Table 4

[0141]

[0142] As shown in Table 4, automatic α learning significantly improves model recognition performance under all three initial α values. When α is fixed, the model performs best with α=0.20; when α is learned automatically, the model with an initial α value of 0.30 yields the optimal results. However, when the α value is too large, the model loses too many action skeleton features and physical structure details, leading to a decrease in recognition accuracy. Meanwhile, these experimental results also verify the effectiveness of injecting LLM semantic knowledge into physical structure information for action recognition tasks.

[0143] Hybrid Adjacency Matrix Visualization Analysis Experiment:

[0144] To qualitatively analyze the impact of different α values ​​on model performance, the physical skeleton adjacency matrix, the joint semantic adjacency matrix, and the mixed adjacency matrix with different initial weights were visualized, as shown in Figures 5(a)-5(g). Specifically, the three subsets of the physical adjacency matrix are as follows: Figures 5(a)-5(c) As shown, the skeleton node exhibits a sparse structure, with most elements being zero, indicating a lack of direct connections between skeleton nodes. In contrast, the semantic adjacency matrix of joints in Figure 5(d) is denser, with most elements being non-zero, reflecting a continuously distributed semantic similarity relationship. Unlike the local joint connection information represented by the physical adjacency matrix, the semantic joint matrix can capture joint functional and overall relationships that extend beyond the scope of physical action connections. Therefore, the fusion of the two enriches the action feature representation of the skeleton data from both local and global joint perspectives.

[0145] From the hybrid adjacency matrix Figures 5(e)-5(g) In the diagram, a dynamic fusion process of physical structure and semantic knowledge information can be observed. As the initial weight α increases from 20% to 40%, the semantic background noise in the matrix gradually increases. Compared to Figure 5(g), Figures 5(e) and 5(f) show a balanced state: the originally sparse physical skeleton is endowed with appropriate semantic association enhancement, which preserves the spatial structural features of the skeleton while introducing action semantic information. This balanced state is consistent with the high action recognition performance achieved by the model when the α value is set to 0.20 and 0.30 in Table 4.

[0146] Fine-grained experimental analysis of the hierarchical temporal feature extraction module:

[0147] To explore the optimal temporal hierarchical scale and hierarchical combination strategy for the model based on the hybrid graph topology learning strategy, fine-grained experimental analysis was conducted. The results are shown in Table 5, which presents the fine-grained ablation experimental results (%) of the hierarchical temporal feature extraction module. Considering training overhead and time cost, this part uniformly uses the X-sub benchmark of the PKU-MMD Ⅰ dataset for experiments.

[0148] Table 5

[0149]

[0150] The experimental results in Table 5 show that the reasonable division of the temporal receptive field has a significant guiding effect on the learning of the spatiotemporal features of complex actions. ① In the horizontal comparison of single-kernel configurations, using a larger receptive field (kernel size 7) can more stably capture long-term temporal dependencies, and its classification performance is relatively better than that of convolutional branches that only focus on local short-term dynamics (kernel sizes 3 and 5); however, when the receptive field is further expanded (kernel sizes 9 and 11), the model performance drops significantly. This phenomenon reflects that the cross-regional spatial joint representation constructed by LLM prior guidance requires a relatively wide but reasonably configured time window to fully learn the continuous evolution of action features. ② In the validation of dual-kernel combinations, it can be observed that the blind parallel connection of multi-scale branches does not necessarily translate into performance gain. For example, when the local short-term branch is paralleled with the mid-range branch (3,5), the model's ability to learn action features decreases; in contrast, the combination of mid-range and global convolution (5,7) shows better complementarity. ③ When using the complete hierarchical three-kernel structure (3,5,7), the model's action recognition accuracy reached a maximum of 98.3%; however, when using larger convolutional kernels to construct the hierarchical three-kernel structure, the model performance declined again, which once again proves that introducing excessively large convolutional kernels will cause the model to ignore detailed action features and affect the overall performance of the model.

[0151] 4. Model complexity analysis experiment:

[0152] To ensure the completeness of model performance evaluation, the model size and computational cost are analyzed using two metrics: the number of model parameters and GFLOPs. Four algorithms were specifically selected for comparison: DS-STGCN, DSDC-GCN, TSGCNeXt, and GAP. The number of model parameters and GFLOPs were calculated based on the original data or open-source programs of the algorithms. The experimental results are shown in Table 6, which presents the results of the analysis of model parameter count and computational cost.

[0153] Table 6

[0154]

[0155] As can be seen, the core network of LLMA-GCN has 14.61M parameters, significantly smaller than current mainstream visual Transformer models. Compared to GAP combined with LLM, LLMA-GCN has a relatively larger number of parameters but fewer GFLOPs. Furthermore, the GAP algorithm requires CLIP for text encoding during training, resulting in significant training overhead. Algorithms without LLM generally have smaller parameter counts and fewer GFLOPs, and lightweighting is typically a key research focus for this type of algorithm. However, when such models introduce deeper network structures or use multiple modal skeleton flows to extract more robust deep multidimensional features, the model complexity increases significantly, as seen in TSGCNeXt. In addition, LLMA-GCN achieves an action recognition inference speed of approximately 25 FPS on a single NVIDIA RTX 3090 GPU, which is sufficient to meet the real-time action inference requirements of consumer-grade graphics cards for data such as surveillance videos.

[0156] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for skeleton action recognition of a large language model, characterized in that, Includes the following steps: The semantic adjacency matrix of joints and the semantic vector of actions are obtained through a large language model, and the physical skeleton adjacency matrix is ​​extracted through skeleton action data. A hybrid adjacency matrix for skeleton actions is constructed using the joint semantic adjacency matrix and the physical skeleton adjacency matrix, which is used to enhance the features of the skeleton actions. A hierarchical visual sequence encoder is constructed, and visual features are obtained through the hierarchical visual sequence encoder; Based on the action semantic vector, the visual features are subjected to visual-semantic feature alignment processing to output the recognition information and prediction results of the skeleton action.

2. The large language model skeleton action recognition method according to claim 1, characterized in that, A hybrid adjacency matrix for skeleton actions is constructed using the joint semantic adjacency matrix and the physical skeleton adjacency matrix, which is used to enhance the features of the skeleton actions, specifically including: A parameter matrix is ​​constructed based on the importance of different physical connections of the skeleton motion. The parameter matrix is ​​then multiplied element-wise with the physical skeleton adjacency matrix to obtain an adaptive adjacency matrix. The hybrid adjacency matrix is ​​obtained based on the adaptive adjacency matrix and the keypoint semantic adjacency matrix.

3. The large language model skeleton action recognition method according to claim 2, characterized in that, Constructing a hierarchical visual sequence encoder, and acquiring visual features through the hierarchical visual sequence encoder, specifically includes: The hierarchical visual sequence encoder comprises three stages, each of which includes three stacked refinement blocks, each of which is based on the large language model.

4. The large language model skeleton action recognition method according to claim 3, characterized in that, Each of the refined blocks includes three parallel graph convolution branches and three parallel dynamic convolution branches; The input features of the refined block are respectively passed through the three graph convolution branches, and then added element by element to obtain feature aggregation; The feature aggregation establishes a residual connection with the input features to obtain intermediate features; The intermediate features are input into the three dynamic convolutional branches respectively, and then concatenated to obtain multi-scale temporal features; The multi-scale temporal features are connected with the input features and the intermediate features through a double residual concatenation to obtain the output features.

5. The large language model skeleton action recognition method according to claim 4, characterized in that, The three dynamic convolutional branches employ 3×1, 5×1, and 7×1 convolutional kernels, respectively.

6. The large language model skeleton action recognition method according to claim 5, characterized in that, The hierarchical visual sequence encoder extracts joint flow data, joint motion flow data, bone flow data, and bone motion flow data of the skeletal movement to obtain the visual features.

7. The large language model skeleton action recognition method according to claim 6, characterized in that, Based on the action semantic vector, the visual features are aligned visually with semantically, and the recognition information and prediction results of the skeleton action are output, specifically including: The visual features are input into a fully connected classifier and a visual feature projector, respectively. The classification logic value is obtained through the fully connected classifier, and the visual feature vector is obtained through the visual feature projector. The action semantic vector is normalized and embedded to obtain the action text prototype; The action text prototype is projected onto a visual-semantic space shared with the visual feature vector using a text feature projector, and the action text feature vector is output. Calculate the cosine similarity between the visual feature vector and the action text feature vector to obtain the text alignment logical value.

8. The large language model skeleton action recognition method according to claim 7, characterized in that, Based on the action semantic vector, the visual features are aligned visually and semantically to output the recognition information and prediction results of the skeleton action, and the process further includes: The classification cross-entropy loss is constructed using the classification logical values; The text alignment cross-entropy loss is constructed using the text alignment logical values; The classification cross-entropy loss and the text alignment cross-entropy loss are weighted to obtain a joint loss, thus completing the visual-semantic feature alignment process.

9. A large language model skeleton action recognition system, executing the large language model skeleton action recognition method as described in any one of claims 1 to 8, characterized in that, include: The hybrid graph topology learning module is used to construct a hybrid adjacency matrix for skeleton actions; The encoding module is used to build a hierarchical visual sequence encoder to obtain visual features; The visual-semantic alignment module is used to perform visual-semantic feature alignment processing on the visual features.

10. An electronic device, characterized in that, include: At least one processor; A memory that is communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: The semantic adjacency matrix of joints and the semantic vector of actions are obtained through a large language model, and the physical skeleton adjacency matrix is ​​extracted through skeleton action data. A hybrid adjacency matrix for skeleton actions is constructed using the joint semantic adjacency matrix and the physical skeleton adjacency matrix, which is used to enhance the features of the skeleton actions. A hierarchical visual sequence encoder is constructed, and visual features are obtained through the hierarchical visual sequence encoder; Based on the action semantic vector, the visual features are subjected to visual-semantic feature alignment processing to output the recognition information and prediction results of the skeleton action.

Images