An action recognition method and system based on fusion of skeletal features and multi-layer entity semantic features

Abstract

The application discloses a motion recognition method and system based on fusion of skeleton features and multi-layer entity semantic features. The method is as follows: firstly, a hierarchical multi-granularity entity recognition (HMER) module is used to process text description of a data set, structured semantics are extracted, and the structured semantics are converted into entity importance distribution; an anatomical correspondence function is used to map semantic importance to skeleton joint attention weight; skeleton features extracted by a graph convolution network are combined with the graph convolution network, visual features are modulated through a semantic feature mechanism, and a visual recognition model is guided to focus on discriminative body parts; and finally, motion recognition is completed through classification loss optimization; the system is used for implementing the motion recognition method based on fusion of skeleton features and multi-layer entity semantic features; and the application can more accurately and efficiently improve motion recognition prediction results.

Description

Technical Field

[0001] This invention relates to the field of video processing and understanding technology, and in particular to an action recognition method and system based on the fusion of skeletal features and multi-layer entity semantic features. Background Technology

[0002] Action recognition technology, as one of the core research directions in the field of computer vision, is an intelligent technology that automatically determines the type of action or behavioral intent by analyzing the posture, motion trajectory, and interaction patterns of the human body in video sequences. Its core lies in the accurate extraction and effective representation of the dynamic features of the human body. This technology has formed a development path of "single-modal feature extraction - multi-modal fusion - semantic enhancement". The current mainstream methods mainly revolve around visual features (such as RGB images and skeletal key points). Among them, skeletal features have become one of the core modalities of action recognition due to their advantages such as resistance to background interference and strong robustness to changes in illumination. Typical technologies include skeletal feature modeling methods based on graph convolutional networks (GCN) (such as ST-GCN and CTR-GCN) and feature enhancement methods based on attention mechanisms.

[0003] The application areas of motion recognition technology have been widely covered, including: sign language interaction, intelligent monitoring and security, human-computer interaction, motion analysis and medical rehabilitation, and video retrieval and content analysis.

[0004] Despite significant progress in action recognition technology, existing methods still suffer from key shortcomings in complex action scenarios (such as fine-grained sign language recognition and similar action differentiation): severe disconnect between vision and semantics, lack of fine-grained semantic information, absence of a mapping mechanism between semantic and visual features, and insufficient support from dedicated datasets. To address these shortcomings, this invention aims to solve the following core technical problems: Constructing a hierarchical semantic representation system for actions, transforming unstructured action descriptions into structured semantic knowledge, clarifying the hierarchical structure of actions such as primary and secondary entities, action location, and spatiotemporal interactions between entities, thus compensating for the lack of semantic information in existing technologies; Establishing a dynamic mapping mechanism between semantic and visual features, transforming hierarchical semantic information into attention-guided signals for skeletal joints, achieving "priority learning of semantically critical parts," and solving the problems of disconnect between visual features and semantics and insufficient model focus; Designing a feature extraction and modulation framework adapted to semantic guidance, integrating semantic constraints into the skeletal feature learning process, strengthening semantically relevant fine-grained features, suppressing irrelevant redundant information, improving the ability to differentiate similar actions, and promoting the practical application of the technology in professional scenarios such as sign language recognition. Summary of the Invention

[0005] To overcome the problems existing in the prior art, the present invention aims to provide an action recognition method and system based on the fusion of skeletal features and multi-layer entity semantic features. This method can deeply integrate hierarchical entity semantic information of actions on the basis of accurately extracting the spatiotemporal features of key skeletal points. By constructing a dynamic mapping mechanism between semantic and visual features, it guides the visual recognition model to focus on key semantic parts and capture fine-grained semantic association details. At the same time, it makes up for the defects of visual and semantic disconnect in the prior art. By combining structured semantic features and robust skeletal features, it achieves more accurate action category differentiation and improves the accuracy and generalization ability of action recognition in complex scenes.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An action recognition method based on fusing skeletal features and multi-layer entity semantic features includes the following steps: Step 1: Obtain the sign language video to be recognized and its corresponding text description; Step 2: Based on the text description obtained in Step 1, extract multi-level entity semantic information through an HMER module that has been pre-trained and has its parameters frozen on a dataset with four-level hierarchical annotations, and generate an entity importance distribution that represents the importance of each entity; wherein, the HMER module is configured to parse out the semantic information of four levels: main entity, secondary entity, action visual features, and spatiotemporal interaction relationship between entities. Step 3: Using a learnable mapping function, the entity importance distribution described in Step 2 is converted into attention weights acting on the joints of the human skeleton. Step 4: Extract skeletal sequence features from the sign language video in Step 1, and use the attention weights described in Step 3 to perform weighted modulation on the skeletal sequence features to obtain semantically enhanced skeletal features. Step 5: Based on the semantically enhanced skeletal features obtained in Step 4, the final sign language action category is output through a classifier.

[0007] The HMER module includes a pre-trained language model and multiple dedicated neural network layers. The pre-trained language model is a text encoder of the CLIP model, which encodes the text into contextual embeddings. The multiple dedicated neural network layers extract four levels of semantic representations: the visual features of the positions of the main entities, secondary entities, and actions, as well as the spatiotemporal interaction relationships between entities. The weighted summation is used to obtain the total HMER loss. After training, the parameters of the HMER module are frozen as a static semantic knowledge base, and forward calculation is continued to output the entity importance distribution.

[0008] The learnable mapping function described in step 3 is a linear projection layer. The obtained entity importance distribution is mapped to skeletal joint attention weights using an anatomical correspondence function. The specific method in step 3 is as follows: anatomical correspondence function... The entity importance vector that distributes entity importance. Mapping to act on Attention weight vector at each skeletal joint This mapping is defined as: in, It is a learnable weight matrix that encodes the mapping relationship between language entities and anatomical joints. It is a learnable bias vector. It is the entity importance vector The dimension; the Softmax function normalizes the output to ensure The weights in the equation constitute an effective probabilistic attention distribution at the key points.

[0009] Step 4, which involves extracting skeletal sequence features from the sign language video, specifically includes: obtaining the skeletal keypoint sequence corresponding to the sign language video, and inputting the skeletal keypoint sequence into a graph convolutional network to obtain skeletal sequence features; the graph convolutional network is HR-Net or CTR-GCN; the skeletal sequence features are weighted and modulated using attention weights, which is achieved by element-wise multiplication.

[0010] The specific method for step 4 is as follows: Step 4.1: Use a pre-trained graph convolutional network to detect key points from each frame of the sign language video to be recognized, and standardize them to obtain initial skeletal features; Step 4.2: The graph convolutional network outputs the initial skeletal features. Based on the attention weights at the human skeleton joints in step 3, the skeletal features output by the graph convolutional network are weighted and modulated using element-wise multiplication: At time step Original feature map The following optimizations have been made: in, It is a learnable parameter matrix used to tune attention to a specific space. The modulation method represents the resulting attention-weighted feature map. .

[0011] The specific method for step 5 is as follows: The visual model is optimized using cross-entropy loss: in, It refers to the number of sign language categories. If it's a genuine label, the correct category is 1; otherwise, it's 0. It is a model for categories The predicted probability is obtained by analyzing the modulated features. Temporal pooling and classification are performed to obtain the data; the data is then input into a classifier to complete action recognition, and the accuracy is improved through loss optimization.

[0012] An action recognition system based on fusion of skeletal features and multi-layer entity semantic features includes: The data acquisition module is used in step 1 to acquire the sign language video to be recognized and its corresponding text description; The HMER semantic extraction module is used in step 2 to extract multi-level entity semantic information from the text description and generate entity importance distribution by using an HMER module that has been pre-trained and has its parameters frozen on a dataset with four levels of hierarchical annotation. The HMER module is configured to parse out the semantic information of four levels: main entity, secondary entity, visual features such as the position of the action, and the spatiotemporal interaction relationship between entities. The semantic-skeleton mapping module is used in step 3 to convert the entity importance distribution into attention weights acting on the joints of the human skeleton using a learnable mapping function. The skeletal feature modulation module is used in step 4 to extract skeletal sequence features from the sign language video and to perform weighted modulation on them using the attention weights to obtain semantically enhanced skeletal features. The action recognition module is used in step 5 to output the final sign language action category based on the semantically enhanced skeletal features.

[0013] An action recognition device based on fused skeletal features and multi-layer entity semantic features includes: Memory, used to store computer programs; A processor, configured to implement the action recognition method based on fused skeletal features and multi-layer entity semantic features as described in any one of claims 1 to 6 when referring to the computer program.

[0014] A computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the program implements the action recognition method based on fused skeletal features and multi-layer semantic features as described in any one of claims 1 to 6.

[0015] Compared with the prior art, the beneficial effects of the present invention are: First, the technical means to achieve semi-automatic annotation: This invention uses a hierarchical multi-granularity entity recognition (HMER) module to encode action text descriptions into contextual embeddings using a pre-trained language model (such as CLIP). This module uses a three-way parallel dedicated Transformer architecture layer to automatically parse the text, extract visual feature information such as the location of main entities, secondary entities, and actions, and combines it with a relationship extraction module to model the spatiotemporal relationships between entities.

[0016] Second, it significantly reduces data preparation costs: This solution achieves text-based annotation transformation, eliminating the need for expensive manual frame-by-frame skeletal point annotation of video frames. The system only requires the original motion video and its corresponding standard text description, and can automatically distill structured semantic knowledge through the HMER module, greatly reducing the time and manpower costs of dataset construction.

[0017] Third, it preserves fine-grained semantic association information: Unlike traditional single-category labels, this dataset fully preserves the subtle semantics of actions through four-level hierarchical annotation (primary entities, secondary entities, visual features such as the position of actions, and spatiotemporal interaction relationships between entities). This multi-granular semantic feature can capture discriminative details such as "finger joint movement trends," effectively solving the ambiguity in the visual representation of similar actions.

[0018] Fourth, semantically guided precise feature modulation: The entity importance distribution generated by the dataset is mapped to skeletal joint attention weights through an anatomical correspondence function. This enables the model to automatically enhance key parts (such as hands and fingers) among the 87 skeletal keypoints in physical space according to semantic logic, and suppress irrelevant background noise, achieving deep fusion of semantic and visual features.

[0019] In summary, compared with existing technologies, this invention significantly improves the accuracy of action recognition based on skeletal sequences through a language prior-driven semantic perception attention mechanism without adding complex sensors or additional annotations. It is particularly suitable for challenging tasks such as semantically dense and subtle sign language understanding, and has outstanding technical advancement and practical application value. Attached Figure Description

[0020] Figure 1 This is a flowchart of the action recognition method based on fused skeletal features and multi-layer entity semantic features as described in an embodiment of the present invention; Figure 2 is a schematic diagram of entity weight and attention mapping according to an embodiment of the present invention. In Figure 2(a), the entity weight distribution extracted by HMER is shown. Figure 2(b) shows the mapping of importance scores at skeletal joints. Figure 2(c) shows the feature attention distribution before semantic guidance. Figure 2(d) shows the feature attention distribution after semantic guidance. Figure 3 is a visualization comparison of the feature distribution t-SNE described in the embodiment of the present invention. In Figure 3(a), the feature distribution does not include hierarchical entity semantic features (HRSS), and Figure 3(b) is the feature distribution including HRSS. Figure 4 is a schematic diagram of the semantic feature modulation process according to an embodiment of the present invention, showing the generation process from semantic prior vector to modulated feature map; Figure 5 is a schematic diagram of the system described in an embodiment of the present invention; Figure 6 is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. Detailed Implementation

[0021] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0022] like Figure 1 As shown, this embodiment provides an action recognition method based on the fusion of skeletal features and multi-layer entity semantic features, including the following steps: Step 1: Obtain the sign language video to be recognized and its corresponding text description; 1.1) Input the constructed hierarchical dataset into HMER, an expert model that parses sign language semantics from text descriptions to capture the complex, hierarchical semantic structure of sign language; 1.2) Use the pre-trained language model CLIP to encode text into context and generate latent representations. Where d is the embedding dimension. Based on these embeddings, a specialized Transformer layer... , , and It is used to extract hierarchical semantic components. The relationship extraction module models the spatial and temporal relationships between entities, such as relative position and sequential actions. here, , and These represent the visual features of the primary entity, secondary entity, and action, respectively. It is a level The number of entities, It is the corresponding feature dimension. Then, the relationships between entities are modeled to generate relational representations. .

[0023] For each level of entity recognition, calculate a cross-entropy loss: in, It is the number of categories, and and They are entities Belongs to level Category True probability and predicted probability.

[0024] Similarly, for the extraction of relationships between entities, the relationship loss is calculated: in, Representing entities and Does it have a relation type, and That is the corresponding predicted probability.

[0025] The total HMER loss is a weighted sum of the cross-entropy losses of each component: in, and These are balancing weights used to control the contribution of each component to the overall training objective. The HMER module is trained independently on a text corpus, effectively distilling structured linguistic knowledge. After convergence, the module's parameters are frozen, transforming it into a static knowledge base used to generate entity-guided attention masks for the visual model in subsequent training phases.

[0026] 1.3) Using a frozen HMER model, an entity importance distribution is derived for each sign language category. This knowledge is formalized as an importance vector. ,in Indicates the first Entities (e.g., hands, faces) are categories The relative importance of these values. These values ​​are hierarchical representations learned from HMER. Extracted from: in, : is the projection function, used to project from the level Extract entity-specific importance from the representation. It is a weighting coefficient that balances the contributions from the main entity, secondary entity, and action visual features.

[0027] Step 2: Based on the text description obtained in Step 1, extract multi-level entity semantic information through an HMER module that has been pre-trained and has its parameters frozen on a dataset with four-level hierarchical annotations, and generate an entity importance distribution that represents the importance of each entity; wherein, the HMER module is configured to parse out the semantic information of four levels: main entity, secondary entity, action visual features, and spatiotemporal interaction relationship between entities. 2.1) Input the constructed hierarchical dataset into HMER, an expert model that parses sign language semantics from text descriptions, in order to capture the complex and hierarchical semantic structure of sign language; 2.2) Use the pre-trained language model CLIP to encode text into context and generate latent representations. Where d is the embedding dimension. Based on these embeddings, a specialized Transformer layer... , , and It is used to extract hierarchical semantic components. The relationship extraction module models the spatial and temporal relationships between entities, such as relative position and sequential actions. here, , and These represent the visual features of action, including the position of the main entity, secondary entities, and actions. It is a level The number of entities, It is the corresponding feature dimension. Then, the relationships between entities are modeled to generate relational representations. .

[0028] For each level of entity recognition, calculate a cross-entropy loss: in, It is the number of categories, and and They are entities Belongs to level Category True probability and predicted probability.

[0029] Similarly, for the extraction of relationships between entities, the relationship loss is calculated: in, Representing entities and Does it have a relation type, and That is the corresponding predicted probability.

[0030] The total HMER loss is a weighted sum of the cross-entropy losses of each component: in, and These are balancing weights used to control the contribution of each component to the overall training objective. The HMER module is trained independently on a text corpus, effectively distilling structured linguistic knowledge. After convergence, the module's parameters are frozen, transforming it into a static knowledge base used to generate entity-guided attention masks for the visual model in subsequent training phases.

[0031] 2.3) Using a frozen HMER model, an entity importance distribution is derived for each sign language category. This knowledge is formalized as an importance vector. ,in Indicates the first Entities (e.g., hands, faces) are categories The relative importance of these values. These values ​​are hierarchical representations learned from HMER. Extracted from: in, : is the projection function, used to project from the level Extract entity-specific importance from the representation. It is a weighting factor that balances the contributions from visual features such as the position of the main entity, secondary entities, and actions.

[0032] Step 3: Using a learnable mapping function, the entity importance distribution described in Step 2 is converted into attention weights acting on the joints of the human skeleton. Anatomical Corresponding Functions Instantiated as a learnable linear projection layer. The function's purpose is to transform the abstract entity importance vector... Transformed into a specific, category-specific entity that acts upon Attention weight vector at each skeletal joint This mapping is defined as: in, It is a learnable weight matrix that encodes the mapping relationship between language entities and anatomical joints. It is a learnable bias vector. It is the entity importance vector The dimension. The Softmax function normalizes the output to ensure... The weights in the equation constitute an effective probabilistic attention distribution at the key points.

[0033] Step 4: Extract skeletal sequence features from the sign language video in Step 1, and use the attention weights described in Step 3 to perform weighted modulation on the skeletal sequence features to obtain semantically enhanced skeletal features. 4.1) A pre-trained HR-Net is used to detect 87 keypoints in each frame of the SLR-500 dataset video. These keypoints are then processed by the CTR-GCN backbone network and normalized to 300 frames to obtain the initial skeletal features.

[0034] 4.2) The graph convolutional network CTR-GCN outputs skeletal features. Based on the attention weights at the human skeleton joints in step 3, the skeletal features output by the graph convolutional network are weighted and modulated using element-wise multiplication. At time step Original feature map The following optimizations have been made: in, It is a learnable parameter matrix used to tune attention to a specific space. This represents element-wise multiplication; the resulting attention-weighted feature map It emphasizes that HMER is identified as key points and actions with important semantics.

[0035] Step 5: Based on the semantically enhanced skeletal features obtained in Step 4, the final sign language action category is output through a classifier. The visual model is optimized using a standard supervised classification objective—cross-entropy loss: in, It refers to the number of sign language categories. It is a true label (1 for the correct category, 0 otherwise). It is a model for categories The predicted probability is obtained by analyzing the modulated features. The results are obtained through temporal pooling and classification. The input classifier performs action recognition, and the accuracy is improved through loss optimization.

[0036] Figure 2 illustrates the entity weight and attention mapping diagram, visually demonstrating the core mechanism of this invention—the dynamic guidance process from semantics to vision; the semantic parsing capability of the Hierarchical Multi-Granularity Entity Recognition (HMER) module; and the feature guidance effect of semantic features. Figure 2(a) shows the entity weight distribution extracted by HMER. The HMER module successfully parses multi-level entities (such as "right hand," "index finger," and "moving upwards") from the sign language text description and quantifies their importance scores, forming a structured entity importance distribution. This distribution is not uniform or random, but closely corresponds to the discriminative semantic elements of the action. Furthermore, through the anatomical correspondence function (Figure 2(b)) mapping the importance scores to skeletal joints, these abstract semantic weights are precisely mapped to the 87 skeletal joints of the human body. For example, when the sign language action emphasizes "index finger pointing," the system automatically assigns high attention weights to multiple joints of the right index finger (such as the fingertip and knuckle), while trunk or leg joints are given extremely low weights. This mapping reflects the fusion of prior knowledge of language and anatomical structure. Figure 2(c) shows the feature attention distribution before semantic guidance, and Figure 2(d) shows the feature attention distribution after semantic guidance. Comparing Figure 2(c) and Figure 2(d), it can be seen that before the introduction of semantic guidance (Figure 2(c)), the model's attention distribution to skeletal joints is relatively diffuse, making it difficult to distinguish between key and non-key parts; while after semantic guidance (Figure 2(d)), the attention is highly concentrated on the semantically key regions (such as the hand) identified by HMER, and the responses of background or irrelevant limb parts are significantly suppressed. This proves that the present invention can effectively achieve "semantic key part priority learning", solve the feature focusing bias problem caused by the visual-semantic disconnect in existing methods, and provide an interpretable and efficient feature enhancement path for fine-grained action recognition.

[0037] Figure 3 shows a visualization comparison of the feature distribution t-SNE described in this invention, used to verify the effect of Hierarchical Entity Semantic Features (HRSS) on improving feature discriminativeness; it also compares the impact of introducing HRSS on the final skeletal feature representation. Figure 3(a) shows the feature distribution without HRSS. Figure 3(a) shows that when only the original skeletal features are used (without HRSS), the feature point distributions of different sign language categories overlap significantly, especially between semantically or morphologically similar action categories (such as "hello" and "goodbye"), where the boundaries are blurred and difficult to separate linearly. However, after introducing the HRSS mechanism proposed in this invention, Figure 3(b) shows the feature distribution with HRSS. The feature clusters of each category exhibit more compact cohesion and clearer inter-class separation. Feature points of similar actions are no longer mixed, but form independent clusters with distinct boundaries. This phenomenon indicates that semantically guided attention modulation not only strengthens the feature expression of discriminative key points, but also effectively suppresses intra-class variation (such as individual gesture amplitude differences) and amplifies inter-class differences (such as different directions of key finger movements).

[0038] like Figure 4 The diagram illustrates the semantic feature modulation process described in this invention, showing the generation flow from semantic prior vectors to modulated feature maps, which are used for the fusion of skeletal features and multi-layer entity semantic features. Figure 4 The specific process of semantic feature modulation in this invention is described in detail. First, the entity importance vector (semantic prior) output by the HMER module is converted into a skeletal joint attention weight vector through a learnable anatomical correspondence function (i.e., linear projection layer + Softmax). This weight vector has a clear physical meaning—each dimension corresponds to the semantic relevance of a joint. Subsequently, this attention weight vector is multiplied element-wise with the original skeletal feature map extracted by the graph convolutional network. This operation is not a simple weighted average, but a spatially adaptive feature recalibration: the feature channels of joints with high semantic importance are amplified, while the feature responses of irrelevant joints are attenuated or even set to zero.

[0039] It is worth noting that the learnable parameter matrix introduced during the modulation process This allows the model to fine-tune anatomical mappings during training, making semantic guidance more aligned with the data distribution of the specific task. The final generated semantically enhanced skeletal feature map ( Figure 4 (On the right) It retains the spatiotemporal dynamic structure of the original skeleton while embedding high-level semantic constraints from the text description, achieving a deep fusion of visual features and linguistic semantics. This mechanism is the key technical support for achieving high-precision action recognition in this invention.

[0040] like Figure 5As shown, an action recognition system based on the fusion of skeletal features and multi-layer entity semantic features is presented. The system includes a multi-layer entity semantic extraction module, a semantic-skeleton mapping module, a semantic-guided feature modulation module, and an action recognition optimization module. The multi-layer entity semantic extraction module uses the CLIP text encoder as the language model, and is equipped with three parallel Transformer sub-modules Φ1, Φ2, and Φ3, which focus on extracting semantic representations of visual features such as the position of primary entities (e.g., right hand, both hands), secondary entities (e.g., thumb, fingers), and the position of actions (e.g., upward, circular motion). Then, the MLP module Ψ models the relationships between entities and outputs the distribution of entity importance. The semantic-skeleton mapping module designs an anatomical correspondence function to map the entity importance vector to the attention weights of 87 skeletal joints. After Softmax normalization, the semantic relevance weight of each joint is obtained. The semantic guidance feature modulation module modulates the initial skeletal features and attention weights element-wise by multiplication, thereby strengthening the semantically important joint features and suppressing the semantically irrelevant joint features. The action recognition optimization module performs temporal pooling on the enhanced skeletal features, inputs them into an MLP classifier to obtain the predicted probability of the action category, calculates the cross-entropy classification loss, and optimizes the parameters of the semantic-skeleton mapping module, the semantic-guided feature modulation module, and the classifier end-to-end.

[0041] It should be noted that the above system is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. This system is the first-person perspective action recognition method applied to the above embodiment.

[0042] A dual-stream global-local action recognition device based on video multi-local extraction includes: a memory for storing a computer program; A processor, configured to execute the computer program, implements the action recognition method based on fused skeletal features and multi-layer entity semantic features as described in any one of claims 1 to 6.

[0043] like Figure 6 As shown, a storage medium stores a program that, when executed by a processor, implements the first-person perspective action recognition method of the above embodiments, specifically as follows: The human key point recognition method is used to extract human key points from the video and select the extracted human key points. Multiple human body parts are then cropped according to the key point positions and used as input for the local feature extraction module. We use the clipping and flipping methods commonly used in the field to preprocess the global and local input data.

[0044] Simulation Experiment Design All experiments were implemented using PyTorch on an NVIDIA A100-40GB GPU, with a fixed random seed to ensure reproducibility. For skeletal feature extraction, a pre-trained HR-Net was used to detect 87 keypoints in each frame of the SLR-500 dataset video. The data was processed by the CTR-GCN backbone network and normalized to 300 frames. The current state-of-the-art results achieved used four streams: joints, joint motion, skeleton, and skeleton motion. Text features were extracted from SLR500-HMER annotations using CLIP's text encoder, generating a dimensional... Entity importance vector Subsequently, this vector is processed through the anatomical correspondence function. Mapped to the skeletal joint space, this function will Projecting a semantic vector of dimension onto On each skeletal joint. The model uses the AdamW optimizer (learning rate = Weight decay = End-to-end training was performed for 100 epochs (batch size = 64), using a cosine annealing scheduler and a 5-epoch linear warm-up. All learnable parameters, including All parameters are initialized using the Kaiming parameter initialization method. The hyperparameters of the HMER loss are set as follows: parameters of the main entities. Secondary entity corresponding parameters , attribute parameters Parameters of the relationship In the calculation of the entity importance vector, the weighting coefficients are set to... , and .

[0045] When evaluating the HMER component, this chapter uses F1 score, precision, and recall metrics to assess the quality of entity recognition. For the sign language recognition task, the Top-1 and Top-5 accuracies are reported.

[0046] Comparison with existing technologies: As shown in Table a, the HESS framework achieves an accuracy of 96.7% on the SLR-500 dataset, surpassing all existing methods in the comparison and demonstrating a significant advantage. This result outperforms the previous best method and shows a clear advantage over other competing methods. This validates the effectiveness of hierarchical semantic modeling through HMER and multi-granularity cross-modal alignment in MCA, strategies that together achieve a more comprehensive modeling of the subtle combinatorial structure of sign language. This method can model entities, attributes, and relationships at multiple granularities, while aligning skeletal features with textual semantics, thus achieving superior discriminative power, especially in distinguishing fine-grained sign language variants that pose a challenge to existing frameworks.

[0047] Table a shows the performance comparison on the SLR-500 dataset. method Accuracy (%) ST-GCN 90.0 SignBERT 94.5 BEST 95.4 Hand-Model-Aware 95.9 MASA 96.3 This patented method 96.7 ablation experiment To evaluate the contribution of each component in the proposed framework, this section presents a series of comprehensive ablation studies. These experiments aim to dissect the effects of the major components and the HMER hierarchical structure. All ablation experiments were conducted on the joint flow mode.

[0048] (1) The influence of hierarchical structure in HMER To evaluate the effectiveness of the hierarchical entity recognition method, experiments were conducted to test the performance of the HMER module at different semantic levels. The HMER module was constructed progressively, starting with only the main entity recognition and gradually adding more fine-grained semantic levels.

[0049] As shown in Table b, each semantic level contributes positively to entity recognition performance. The F1 score for primary entity recognition alone reaches 90.2%. When secondary entities are added, the overall F1 score improves to an average of 91.5% across both levels. Adding attributes further improves performance to 92.8%. The complete HMER configuration, including all four levels, achieves a top average F1 score of 93.3%, demonstrating the value of modeling a complete hierarchy of action semantics.

[0050] Table b. Impact of different hierarchical levels on HMER entity recognition performance HMER components Average F1 score (%) only the main entity 90.2 +Secondary Entities 91.5 +Visual features 92.8 +Relationship (Complete HMER) 93.3 This experiment demonstrates that the hierarchical entity recognition method effectively captures the multi-layered semantic structure of sign language descriptions, laying a solid foundation for the subsequent semantic scaffolding process.

[0051] (2) Effectiveness of semantic scaffold components To investigate the contribution of different components in the semantic scaffolding mechanism to the final recognition performance, experiments were conducted using semantic guidance at different levels. Specifically, the experiments examined how the entity-guided attention mechanism performs when it utilizes different subsets of semantic information extracted by HMER.

[0052] Table c. Effectiveness of different semantic components in entity-guided attention mechanism Semantic guidance component Top-1 accuracy (%) Top-5 accuracy (%) Guidance for primary entities only 94.5 97.8 +Secondary Entity Guidance 94.8 98.1 +Visual Feature Guidance 95.1 98.3 +Relationship guidance (complete guidance) 95.4 98.5 The results in Table c demonstrate the progressive benefits of incorporating richer semantic information into attention guidance.

[0053] Using only primary entity information to guide visual attention already improved baseline accuracy to 94.5%. This was further enhanced when secondary entity information was also integrated into the entity importance vector. In the middle stage, the accuracy further increased to 94.8%. Adding attribute guidance brought an additional improvement, with the accuracy reaching 95.1%. Finally, the complete semantic construction method that integrates inter-entity relationship information achieved a maximum accuracy of 95.4%.

[0054] These results highlight the importance of comprehensive semantic understanding in guiding visual attention. By integrating fine-grained semantic details about attributes and relationships between entities, the model is able to more accurately focus on the most discriminative skeletal joints and movements in each sign language category, resulting in superior recognition performance.

[0055] (3) Performance of HMER in entity recognition In addition to its impact on recognition accuracy, this section also evaluates the performance of the HMER component itself on the Sign Language Named Entity Recognition (NER) task. Table d shows the precision, recall, and F1 score for each entity level.

[0056] Performance of the HMER component in semantic entity recognition task. Entity level Accuracy (%) Recall rate (%) F1 score (%) Main Entities 95.3 94.8 95.0 Secondary entities 92.7 91.9 92.3 Visual features 89.6 88.2 88.9 relation 87.4 86.3 86.8 Experimental results show that HMER can effectively identify semantic components at different levels. Specifically, the recognition performance of high-level semantic units such as main entities is significantly better than that of fine-grained semantic details such as relations. This result is consistent with the theoretical expectations of hierarchical semantic modeling. The structured semantic recognition capability provided by HMER lays an important foundation for the reliability of the entire framework.

[0057] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0058] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An action recognition method based on fused skeletal features and multi-layer entity semantic features, characterized in that, Includes the following steps: Step 1: Obtain the sign language video to be recognized and its corresponding text description; Step 2: Based on the text description obtained in Step 1, extract multi-level entity semantic information through an HMER module that has been pre-trained and has its parameters frozen on a dataset with four-level hierarchical annotations, and generate an entity importance distribution that represents the importance of each entity; wherein, the HMER module is configured to parse out the semantic information of four levels: main entity, secondary entity, action visual features, and spatiotemporal interaction relationship between entities. Step 3: Using a learnable mapping function, the entity importance distribution described in Step 2 is converted into attention weights acting on the joints of the human skeleton. Step 4: Extract skeletal sequence features from the sign language video in Step 1, and use the attention weights described in Step 3 to perform weighted modulation on the skeletal sequence features to obtain semantically enhanced skeletal features. Step 5: Based on the semantically enhanced skeletal features obtained in Step 4, the final sign language action category is output through a classifier.

2. The action recognition method based on fused skeletal features and multi-layer entity semantic features as described in claim 1, characterized in that, The HMER module includes a pre-trained language model and multiple dedicated neural network layers. The pre-trained language model is a text encoder of the CLIP model, which encodes the text into contextual embeddings. The multiple dedicated neural network layers extract four levels of semantic representations: the visual features of the positions of the main entities, secondary entities, and actions, as well as the spatiotemporal interaction relationships between entities. The weighted summation is used to obtain the total HMER loss. After training, the parameters of the HMER module are frozen as a static semantic knowledge base, and forward calculation is continued to output the entity importance distribution.

3. The action recognition method based on fused skeletal features and multi-layer entity semantic features as described in claim 1, characterized in that, The learnable mapping function described in step 3 is a linear projection layer. The obtained entity importance distribution is mapped to skeletal joint attention weights using an anatomical correspondence function. The specific method in step 3 is as follows: anatomical correspondence function... The entity importance vector that distributes entity importance. Mapping to act on Attention weight vector at each skeletal joint This mapping is defined as: in, It is a learnable weight matrix that encodes the mapping relationship between language entities and anatomical joints. It is a learnable bias vector. It is the entity importance vector The dimension; the Softmax function normalizes the output to ensure The weights in the equation constitute an effective probabilistic attention distribution at the key points.

4. The action recognition method based on fused skeletal features and multi-layer entity semantic features as described in claim 1, characterized in that, Step 4, which involves extracting skeletal sequence features from the sign language video, specifically includes: obtaining the skeletal keypoint sequence corresponding to the sign language video, and inputting the skeletal keypoint sequence into a graph convolutional network to obtain skeletal sequence features; the graph convolutional network is HR-Net or CTR-GCN; the skeletal sequence features are weighted and modulated using attention weights, which is achieved by element-wise multiplication.

5. The action recognition method based on fused skeletal features and multi-layer entity semantic features as described in claim 1 or 4, characterized in that, The specific method for step 4 is as follows: Step 4.1: Use a pre-trained graph convolutional network to detect key points from each frame of the sign language video to be recognized, and standardize them to obtain initial skeletal features; Step 4.2: The graph convolutional network outputs the initial skeletal features. Based on the attention weights at the human skeleton joints in step 3, the skeletal features output by the graph convolutional network are weighted and modulated using element-wise multiplication: At time step Original feature map The following optimizations have been made: in, It is a learnable parameter matrix used to tune attention to a specific space. The modulation method represents the resulting attention-weighted feature map. .

6. The action recognition method based on fused skeletal features and multi-layer entity semantic features as described in claim 1, characterized in that, The specific method for step 5 is as follows: The visual model is optimized using cross-entropy loss: in, It refers to the number of sign language categories. If it's a genuine label, the correct category is 1; otherwise, it's 0. It is a model for categories The predicted probability is obtained by analyzing the modulated features. Temporal pooling and classification are performed to obtain the data; the data is then input into a classifier to complete action recognition, and the accuracy is improved through loss optimization.

7. An action recognition system based on fused skeletal features and multi-layer entity semantic features, characterized in that, To implement the method according to any one of claims 1 to 6, comprising: The data acquisition module is used in step 1 to acquire the sign language video to be recognized and its corresponding text description; The HMER semantic extraction module is used in step 2 to extract multi-level entity semantic information from the text description and generate entity importance distribution by using an HMER module that has been pre-trained and has its parameters frozen on a dataset with four levels of hierarchical annotation. The HMER module is configured to parse out the semantic information of four levels: main entity, secondary entity, visual features such as the position of the action, and the spatiotemporal interaction relationship between entities. The semantic-skeleton mapping module is used in step 3 to convert the entity importance distribution into attention weights acting on the joints of the human skeleton using a learnable mapping function. The skeletal feature modulation module is used in step 4 to extract skeletal sequence features from the sign language video and to perform weighted modulation on them using the attention weights to obtain semantically enhanced skeletal features. The action recognition module is used in step 5 to output the final sign language action category based on the semantically enhanced skeletal features.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the action recognition method based on fused skeletal features and multi-layer entity semantic features as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the action recognition method based on fused skeletal features and multi-layer entity semantic features as described in any one of claims 1 to 7.

Images