Method and system for semantic augmentation of a pre-trained sign language understanding framework based on bone

Abstract

The application provides a method and system for enhancing a pre-training sign language understanding framework based on bone semantics, and relates to the technical field of sign language recognition, wherein the method comprises: obtaining sign language video data, paired bone sequences and text data thereof; extracting bone key point sequences from the sign language video data and modeling to form bone features, and performing word segmentation on the text; in a pre-training stage, inputting the bone features and the segmented text into a fusion network to generate bidirectional enhanced features, and obtaining global and local similarities through double-level semantic alignment; calculating a contrast loss based on the similarities, and obtaining a hierarchical loss by balancing parameters to coordinate weights; performing a matching task and a language modeling task to obtain corresponding losses, weighting and combining the three types of losses into a pre-training total loss, and adjusting parameters to complete training; and finally, in a fine-tuning stage, optimizing part of the parameters in combination with specific task types to realize enhanced understanding of sign language semantics. The application improves the accuracy of sign language understanding.

Description

Technical Field

[0001] This application relates to the field of sign language recognition technology, and in particular to a method and system for a skeleton-based semantic enhancement pre-trained sign language understanding framework. Background Technology

[0002] Skeletal data-based sign language understanding technology identifies the meaning of gestures by analyzing the movement trajectories of key points on the human body. This type of method has broad prospects in building barrier-free communication environments and intelligent human-computer interaction systems. Furthermore, by extracting skeletal information from videos, background interference can be avoided, allowing focus on the movement characteristics of core parts such as hands and body, providing a technological foundation for cross-language sign language translation and educational applications.

[0003] In existing technologies, skeletal sign language understanding methods typically employ temporal modeling networks to process joint coordinate sequences and combine them with textual information for cross-modal learning. For example, some schemes use graph convolutional networks to capture the spatial relationships between joints and then use recurrent neural networks to model temporal dynamics; other methods introduce attention mechanisms to associate visual features with textual descriptions and use contrastive learning during the pre-training phase to bridge the feature distance between the two modalities.

[0004] However, these methods have limitations in the collaborative modeling of global semantics and local details, affecting the accuracy of the models in understanding complex sign language statements. Therefore, existing technologies suffer from insufficient depth in understanding sign language semantics. Summary of the Invention

[0005] This application provides a method and system for pre-training a sign language understanding framework based on skeleton semantic enhancement, in order to solve the problems of low accuracy and poor semantic coherence in the prior art of sign language understanding.

[0006] To address the aforementioned technical problems, in a first aspect, this application provides a method for pre-training a sign language understanding framework based on skeleton semantic enhancement, comprising:

[0007] Acquire sign language video data, skeletal sequences, and text data paired with the skeletal sequences;

[0008] The skeletal key point sequence is extracted from the sign language video data, and the skeletal key point sequence is modeled using a spatial-temporal graph convolutional network to form skeletal features. The text data is then segmented into words.

[0009] During the pre-training phase, the skeletal features and the segmented text data are input into the early fusion network for sign language perception. Through the cross-attention mechanism, visual features for text guidance and text features for visual guidance are generated. The visual features and text features are then subjected to two-level semantic alignment to determine global similarity and local similarity.

[0010] Based on the global similarity and the local similarity, the corresponding global contrast loss and local contrast loss are determined. The weights of the global contrast loss and the local contrast loss are coordinated by a preset balance parameter to obtain the hierarchical loss.

[0011] Based on the hierarchical loss, a sign language and text matching task and a language modeling task are performed to obtain the matching loss and the language modeling loss respectively. The hierarchical loss, the matching loss and the language modeling loss are weighted and combined to obtain the total pre-training loss. Based on the total pre-training loss, the model parameters are adjusted collaboratively to complete the semantic enhancement pre-training process.

[0012] During the fine-tuning phase, based on the adjusted model parameters and in conjunction with the target task type, some parameters of the semantic enhancement pre-trained sign language understanding framework are fine-tuned to achieve enhanced understanding of sign language semantics.

[0013] Optionally, the step of inputting the skeletal features and the segmented text data into the early fusion network for sign language perception, generating visual features for text guidance and text features for visual guidance through a cross-attention mechanism, and performing two-level semantic alignment on the visual features and the text features to determine global similarity and local similarity includes:

[0014] The skeletal features and the segmented text data are input into the early fusion network for sign language perception, wherein the early fusion network for sign language perception includes a multi-layered sign language encoder and a multi-layered text encoder.

[0015] In each predetermined layer of the sign language encoder, a cross-attention mechanism is used to calculate the first attention weight from text to vision, and visual features for text guidance are generated based on the first attention weight.

[0016] In each predetermined layer of the text encoder, a cross-attention mechanism is used to calculate the second attention weight from vision to text, and visually guided text features are generated based on the second attention weight.

[0017] Global semantic alignment and local semantic alignment are performed on the visual features and the text features respectively to generate corresponding global similarity and local similarity.

[0018] Optionally, the step of performing global semantic alignment and local semantic alignment on the visual features and the text features respectively to generate corresponding global similarity and local similarity includes:

[0019] The visual features are added to the original visual features of the current layer to obtain enhanced visual features, and the text features are added to the original text features of the current layer to obtain enhanced text features.

[0020] The enhanced visual features and the enhanced text features are respectively input into the next layer of the sign language encoder and the text encoder for processing, until the processing of all predetermined layers is completed;

[0021] Extract the first type of token from the enhanced visual features output from the last layer of the sign language encoder, and extract the second type of token from the enhanced text features output from the last layer of the text encoder;

[0022] The first type of token and the second type of token are mapped to the shared semantic space through their respective projection heads. The similarity of the corresponding features of the first type of token and the second type of token after mapping in the shared semantic space is calculated by using dot product to obtain the global similarity.

[0023] Local similarity is generated based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder.

[0024] Optionally, generating local similarity based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder includes:

[0025] The enhanced text features output from the last layer of the sign language encoder and the enhanced visual features output from the last layer of the text encoder are input into a preset aggregator. Multiple word tokens are extracted from the enhanced text features output from the last layer of the text encoder, and multiple visual feature tokens are extracted from the enhanced visual features output from the last layer of the sign language encoder.

[0026] Determine the clustering index for each word token;

[0027] Based on the clustering index, the feature vectors of word tokens belonging to the same clustering result range are merged to generate semantic clustering results;

[0028] Calculate the cosine similarity between each visual feature token and all semantic clustering results, and select the maximum cosine similarity corresponding to each visual feature token;

[0029] The local similarity is obtained by weighted summation of the maximum cosine similarity corresponding to all visual feature tokens.

[0030] Optionally, the step of extracting skeletal keypoint sequences from the sign language video data and modeling the skeletal keypoint sequences using a spatial-temporal graph convolutional network to form skeletal features includes:

[0031] Obtain the skeletal key point sequence of the target area from the sign language video data;

[0032] For the skeletal keypoint sequence of each part, a space-time graph convolutional network is used to model the joint relationships in the spatial dimension and the motion trajectory in the temporal dimension, respectively, to obtain the spatiotemporal dynamic features of each part.

[0033] The spatiotemporal dynamic features of each part are mapped to a unified dimensional space through linear transformation to generate corresponding compact features;

[0034] Connect the compact features of all parts along the feature dimension to form skeletal features.

[0035] Optionally, based on the hierarchical loss, the sign language and text matching task and the language modeling task are performed to obtain the matching loss and the language modeling loss respectively. The hierarchical loss, the matching loss, and the language modeling loss are then weighted and combined to obtain the pre-training total loss. Based on the pre-training total loss, the model parameters are collaboratively adjusted, including:

[0036] Based on the hierarchical loss, a matching task is performed in the target matching path, wherein the enhanced text features output from the last layer of the text encoder and the enhanced visual features output from the last layer of the sign language encoder are input into the multi-layer cross-attention module for deep interaction to generate matching probabilities, and the matching loss is calculated based on the matching probabilities.

[0037] Simultaneously, a language modeling task is performed in the target modeling path, wherein the enhanced text features output from the last layer of the text encoder are input into a multi-layer self-attention module for autoregressive modeling to generate text token prediction results, and the language modeling loss is calculated based on the text token prediction results.

[0038] The hierarchical loss, the matching loss, and the language modeling loss are linearly combined according to the preset first weight coefficient, second weight coefficient, and third weight coefficient to obtain the total pre-training loss;

[0039] Based on the pre-trained total loss, all trainable parameters in the sign language perception early fusion network, the target matching path, and the target modeling path are simultaneously adjusted.

[0040] All trainable parameters are iteratively updated multiple times until the total pre-training loss reaches a preset convergence threshold.

[0041] Optionally, the step of determining the corresponding global contrast loss and local contrast loss based on the global similarity and the local similarity, and coordinating the weights of the global contrast loss and the local contrast loss through a preset balance parameter to obtain the hierarchical loss, includes:

[0042] Based on the global similarity, the first global contrast loss from sign language to text and the second global contrast loss from text to sign language are calculated respectively.

[0043] Based on the local similarity, the first local contrast loss from sign language to text and the second local contrast loss from text to sign language are calculated respectively.

[0044] The first global contrast loss and the second global contrast loss are combined into a global contrast loss, and the first local contrast loss and the second local contrast loss are combined into a local contrast loss.

[0045] The global contrast loss and the local contrast loss are weighted according to preset balance parameters.

[0046] The weighted global contrast loss is added to the weighted local contrast loss to obtain the hierarchical loss.

[0047] Secondly, this application provides a system for a skeleton-based semantic enhancement pre-trained sign language understanding framework, comprising:

[0048] The acquisition module is used to acquire sign language video data, skeletal sequences, and text data paired with the skeletal sequences;

[0049] The extraction module is used to extract the skeletal key point sequence from the sign language video data, model the skeletal key point sequence using a spatial-temporal graph convolutional network to form skeletal features, and perform word segmentation on the text data.

[0050] The input module is used to input the skeletal features and the segmented text data into the early fusion network for sign language perception during the pre-training stage. Through the cross-attention mechanism, it generates visual features for text guidance and text features for visual guidance, and performs two-level semantic alignment on the visual features and the text features to determine global similarity and local similarity.

[0051] The determination module is used to determine the corresponding global contrast loss and local contrast loss based on the global similarity and the local similarity, and to coordinate the weights of the global contrast loss and the local contrast loss through a preset balance parameter to obtain the hierarchical loss;

[0052] The combination module is used to perform sign language and text matching tasks and language modeling tasks based on the hierarchical loss, obtain matching loss and language modeling loss respectively, and weight and combine the hierarchical loss, the matching loss and the language modeling loss to obtain the pre-training total loss. Based on the pre-training total loss, the model parameters are coordinated to complete the semantic enhancement pre-training process.

[0053] The fine-tuning module is used during the fine-tuning phase to fine-tune some parameters of the semantic enhancement pre-trained sign language understanding framework based on the adjusted model parameters and in combination with the target task type, so as to achieve enhanced understanding of sign language semantics.

[0054] Thirdly, this application provides an electronic device, comprising:

[0055] Memory, used to store computer programs;

[0056] A processor, used to execute the computer program, implements the steps of a method for a skeleton-based semantic enhancement pre-trained sign language understanding framework as described in the first aspect above.

[0057] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the method for a skeleton-based semantic enhancement pre-trained sign language understanding framework as described in the first aspect above.

[0058] The technical solution provided in this application has the following beneficial effects:

[0059] First, by acquiring sign language video data and its paired skeletal sequences and text data, a complete input source for multimodal collaborative learning is provided for the model, ensuring the integrity of the data foundation. Then, skeletal keypoint sequences are extracted from the videos and modeled using a part-specific network, which can accurately capture the motion features of different body parts and form a more expressive skeletal feature representation. Furthermore, in the pre-training stage, after fusing the skeletal features with the segmented text input into the network, a cross-attention mechanism is used to achieve deep interaction between visual and textual information, enabling the two modalities to mutually enhance semantic expression. Next, features are aligned semantically at two levels, considering both the overall semantic matching degree and the correspondence of local details, thereby establishing a more comprehensive cross-modal association. Then, a contrastive loss is calculated based on similarity, and the weights are coordinated by balancing parameters, enabling the model to adaptively adjust the learning intensity of global and local features. Subsequently, matching and language modeling tasks are performed to further enhance the cross-modal alignment capability and language generation quality. Finally, in the fine-tuning stage, parameters are optimized in combination with specific tasks, enabling the pre-trained knowledge to be effectively transferred to different sign language understanding scenarios, ultimately improving the model's ability to parse complex sign language semantics.

[0060] Furthermore, in the sign language encoder of the early fusion network for sign language perception, text features are used to apply attention weighting to visual features, generating visual representations that contain linguistic semantics. Simultaneously, visual features are introduced into the text encoder to apply attention adjustment to the text, generating text representations that contain action information. Then, semantic alignment calculations are performed simultaneously on these two enhanced features at both global and local levels to obtain global similarity reflecting the overall semantic matching degree and local similarity capturing the fine-grained corresponding values. This process establishes a close association between visual action features and linguistic semantic features at multiple levels, ensuring both the consistency of overall semantics and the accuracy of local details, thereby improving the integrity and accuracy of cross-modal semantic alignment in sign language understanding.

[0061] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 A flowchart illustrating a method for pre-training a skeleton-based semantic enhancement sign language understanding framework, provided for embodiments of this application;

[0064] Figure 2 A schematic diagram illustrating a specific implementation of a skeleton-based semantic enhancement pre-training sign language understanding framework provided in this application embodiment;

[0065] Figure 3 This is a schematic diagram of the structure of a skeleton-based semantic enhancement pre-trained sign language understanding framework provided in an embodiment of this application. Detailed Implementation

[0066] In the development of sign language understanding technology based on skeletal data, existing methods face a key challenge when dealing with complex sign language statements: when establishing the connection between visual actions and linguistic semantics, the model often struggles to simultaneously consider the correspondence between the overall context and local details. This limitation results in the model having limited ability to distinguish between expressions containing similar gestures but different semantics, thus affecting the accuracy of understanding in practical applications.

[0067] To address this challenge, this application proposes a skeleton-based semantic enhancement pre-training sign language understanding framework. Its core lies in constructing a deep association between visual actions and linguistic semantics through a multi-layered, progressive cross-modal fusion and alignment mechanism. Specifically, this method first introduces an early fusion network for sign language perception during the pre-training stage, achieving bidirectional semantic injection of visual and textual features through cross-attention. Then, a two-layer semantic alignment mechanism is established to simultaneously process global context matching and local action-semantic correspondence. Finally, multi-task collaborative optimization balances the learning intensity of semantic features at different granularities. This layered and progressive approach enables the model to simultaneously capture the overall semantic framework and subtle differences in sign language expressions, thereby improving the ability to distinguish complex sign language while maintaining semantic coherence. This effectively solves the problem of insufficient depth in sign language semantic understanding in existing technologies, providing a technical foundation for building a more accurate sign language understanding system.

[0068] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] The core of this application is to provide a method for pre-training a sign language understanding framework based on skeleton semantic enhancement, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0070] Step 101: Obtain sign language video data, skeletal sequence, and text data paired with the skeletal sequence.

[0071] In step 101, the sign language video data refers to a continuous image sequence containing a person performing sign language actions; the skeletal sequence is the motion trajectory of key points of the human body extracted from the sign language video, which includes the spatial coordinates of the left hand, right hand, body and face at each moment; the text data is a language description that is semantically aligned with the skeletal sequence, which is the words or symbols corresponding to the sign language in the pre-training stage, and the task-specific labels, such as words, sequences or natural language sentences, in the fine-tuning stage.

[0072] In this embodiment of the application, raw sign language video data is acquired by a data acquisition device, and skeletal key points of various parts of the human body are extracted from the video using a pose estimation algorithm to construct a temporal skeletal sequence. At the same time, text description data corresponding to the content of each skeletal sequence is acquired to form multimodal training samples.

[0073] Step 102: Extract the skeletal key point sequence from the sign language video data, model the skeletal key point sequence using a spatial-temporal graph convolutional network to form skeletal features, and perform word segmentation on the text data.

[0074] This involves segmenting the text data into basic units that the text encoder can recognize; for example, the English word "curiosity" is split into "curios" and "ity", and the Chinese word "backpack" is split into "back" and "bag".

[0075] In this embodiment, step 102 includes the following process:

[0076] Step 1021: Obtain the skeletal key point sequence of the target part from the sign language video data.

[0077] In step 1021, the target body includes four parts: left hand, right hand, body, and face. The skeletal key point sequence refers to the sequence of coordinates of human joints detected from video frames on a plane in chronological order. The left hand part contains 21 key points, the right hand part contains 21 key points, the body part contains 9 key points, and the face part contains 18 key points. Each key point is represented by horizontal and vertical coordinate values.

[0078] In this embodiment of the application, sign language video data is processed frame by frame using a pose estimation model to detect the coordinates of key points of four parts of the image: left hand, right hand, body and face, and organize them into four independent skeletal key point sequences in chronological order.

[0079] In practical applications, for a sign language video with a length of 128 frames, after processing by the pose estimation model, the shape of the skeletal keypoint sequence of the left hand is 128×21×2, the sequence of the right hand is 128×21×2, the sequence of the body is 128×9×2, and the sequence of the face is 128×18×2.

[0080] Step 1022: For the skeletal keypoint sequence of each part, a space-time graph convolutional network is used to model the joint relationship in the spatial dimension and the motion trajectory in the temporal dimension to obtain the spatiotemporal dynamic features of each part.

[0081] In step 1022, the spatial-temporal graph convolutional network refers to a graph convolutional neural network designed separately for each body part. The network includes a spatial graph convolutional module and a temporal convolutional module. The spatial graph convolutional module is used to model the spatial dependencies between joints at the same time, and the temporal convolutional module is used to capture the motion change patterns of joints at different times.

[0082] Spatiotemporal dynamic features refer to composite feature representations extracted from skeletal keypoint sequences through a space-time graph convolutional network. These features simultaneously encompass the structural relationships of human joints in the spatial dimension and the movement patterns in the temporal dimension, comprehensively describing the continuous movement patterns of various body parts during sign language execution. Furthermore, by retaining only spatiotemporal dynamic features, storage requirements are reduced by an order of magnitude compared to existing storage methods. For example, the loading time for multimodal training samples is reduced from 455.35ms to 9.58ms.

[0083] In this embodiment, the skeletal key point sequence of each part is input into the corresponding spatial-temporal graph convolutional network. First, the spatial features between joints are extracted through the spatial graph convolutional layer, and then the temporal features of the motion trajectory are captured through the temporal convolutional layer. Finally, the spatiotemporal dynamic features of each part are output.

[0084] In practical applications, a 128×21×2 keypoint sequence of the left hand is processed by a spatial-temporal graph convolutional network to obtain a 128×64 spatiotemporal dynamic feature, where 128 represents the time dimension and 64 represents the feature dimension.

[0085] Step 1023: Map the spatiotemporal dynamic features of each part to a unified dimensional space through linear transformation to generate corresponding compact features.

[0086] In step 1023, compact features refer to the feature representation obtained by mapping high-dimensional spatiotemporal dynamic features to a unified low-dimensional space through linear transformation. Such features have smaller dimensions while retaining the original motion semantic information, which facilitates unified processing and subsequent fusion operations between features of different parts.

[0087] In this embodiment, the spatiotemporal dynamic features of the left hand, right hand, body and face are each mapped to 32 dimensions through independent linear transformation layers to generate four corresponding compact features.

[0088] In practical applications, the 128×64 spatiotemporal dynamic features of the left hand are transformed into 128×32 compact features after linear transformation, and the other three parts are transformed in the same dimension.

[0089] Step 1024: Connect the compact features of all parts along the feature dimension to form skeletal features.

[0090] In step 1024, skeletal features refer to the comprehensive feature representation formed by connecting the compact features of all body parts along the feature dimension. This feature integrates the motion information of multiple parts such as the left hand, right hand, body and face, and constitutes a unified feature representation describing complete sign language movements.

[0091] In this embodiment of the application, the 128×32 compact features of the four parts of the left hand, right hand, body and face are spliced ​​together in the feature dimension to obtain the 128×128 skeletal features, where 128 represents the time step and 128 represents the comprehensive feature dimension.

[0092] For example, first input the key point sequences of the four parts. ,in, This represents the original skeletal input data for the p-th body part, where L is the sequence length. This represents the number of key points for the p-th body part. Let O be the number of key points in the body part, and O be the input feature dimension of each key point. Then, the spatiotemporal dynamic features of each body part are obtained, and then they are uniformly mapped into compact features through a linear transformation. Where D is the projection dimension; finally, the compact features of all parts are stitched together to form the skeletal features. ,in This represents the total feature dimension after concatenation.

[0093] In practical applications, the skeletal features formed by stitching together the compact features of the four parts contain motion information of all key parts during sign language execution, providing a complete visual representation of the action for subsequent semantic understanding.

[0094] This application constructs skeletal features containing multi-part collaborative information through part-specific modeling and feature unification processing, providing a rich visual action representation foundation for cross-modal semantic alignment.

[0095] Step 103: In the pre-training stage, the skeletal features and the segmented text data are input into the early fusion network for sign language perception. Through the cross-attention mechanism, visual features for text guidance and text features for visual guidance are generated. The visual features and text features are then subjected to two-level semantic alignment to determine global similarity and local similarity.

[0096] In this embodiment, step 103 includes the following process, such as... Figure 2 As shown:

[0097] Step 1031: Input the skeletal features and the segmented text data into the early fusion network for sign language perception, wherein the early fusion network for sign language perception includes a multi-layered sign language encoder and a multi-layered text encoder.

[0098] In step 1031, the sign language perception early fusion network refers to a specially designed cross-modal feature fusion architecture. This network includes a multi-layer sign language encoder and a multi-layer text encoder. The sign language encoder processes skeletal features, and the text encoder processes the segmented text data. For example, the sign language encoder performs forward propagation on the input skeletal features S to obtain the visual features of the i-th layer. Simultaneously, the text encoder performs forward propagation on the segmented text data to obtain the text features of the i-th layer. .

[0099] In this embodiment of the application, the 128×128 skeletal features obtained in step 102 are input into the sign language encoder, and the segmented text data is input into the text encoder. Both encoders adopt a multi-layer Transformer structure and set cross-modal connection points in the last few layers.

[0100] In practical applications, skeletal features are processed by a 6-layer sign language encoder, and text data is processed by a 6-layer text encoder. The last two layers of both encoders participate in cross-modal fusion.

[0101] Step 1032: In each predetermined layer of the sign language encoder, the first attention weight from text to vision is calculated using a cross-attention mechanism, and visual features for text guidance are generated based on the first attention weight.

[0102] In step 1032, the first attention weight refers to the distribution of attention of text features to visual features, and the text-guided visual features refer to the visual feature representations that incorporate text semantic information.

[0103] In this embodiment, in the last two layers of the sign language encoder, the output features of the corresponding layer of the text encoder are used as queries, and the current visual features of the sign language encoder are used as keys and values. A first attention weight is calculated through cross-attention, and then the visual features are weighted and fused based on this weight to generate the visual features for text guidance. For example, by calculating... right Cross-attention to generate visual features for text guidance .

[0104] In practical applications, in continuous sign language recognition tasks, the 5th and 6th layers of the sign language encoder participate in fusion, and the text features calculate attention weights for the 128×128 visual features to generate visual features of text guidance of the same dimension.

[0105] Step 1033: In each predetermined layer of the text encoder, a second attention weight from vision to text is calculated using a cross-attention mechanism, and visually guided text features are generated based on the second attention weight.

[0106] In step 1033, the second attention weight refers to the distribution of attention of visual features to text features, and the visually guided text features refer to the text feature representation that incorporates visual action information.

[0107] In this embodiment, in the last two layers of the text encoder, the output features of the corresponding layer of the sign language encoder are used as the query, and the current text features of the text encoder are used as the key and value. A second attention weight is calculated through cross-attention, and then the text features are weighted and fused based on this weight to generate visually guided text features. For example, the calculation... right Cross-attention to generate visually guided text features .

[0108] In practical applications, layers 5 and 6 of the text encoder participate in the fusion, and visual features are used to calculate attention weights on text features to generate visually guided text features that maintain the sequence length and dimension of the original text features.

[0109] Step 1034: Perform global semantic alignment and local semantic alignment on the visual features and the text features respectively to generate corresponding global similarity and local similarity.

[0110] Step 1034 may specifically include the following steps:

[0111] A1: Add the visual features to the original visual features of the current layer to obtain enhanced visual features, and add the text features to the original text features of the current layer to obtain enhanced text features.

[0112] In step A1, the original visual features of the current layer refer to the visual feature representation that the sign language encoder has before performing cross-modal attention interaction when processing the current layer. It is calculated by inputting the output features of the previous layer into the self-attention module and feedforward neural network of the current layer.

[0113] The original text features of the current layer refer to the text feature representation that the text encoder has when processing the current layer before performing cross-modal attention interaction. It is calculated by inputting the output features of the previous layer into the self-attention module and feedforward neural network of the current layer.

[0114] Enhanced visual features refer to new features that are fused with original visual features and text-guided visual features through residual connections. Enhanced text features refer to new features that are fused with original text features and visually guided text features through residual connections.

[0115] In this embodiment of the application, the visual features of the text guidance generated in step 1032 are added element-wise to the original visual features of the current layer to obtain enhanced visual features; at the same time, the text features of the visual guidance generated in step 1033 are added element-wise to the original text features of the current layer to obtain enhanced text features.

[0116] In practical applications, the visual features of the text guidance are added to the original visual features and then fed into the next layer of the sign language encoder for further processing. The features of the two modalities continuously interact and enhance each other during the multi-layer encoding process.

[0117] A2: Input the enhanced visual features and the enhanced text features into the next layer of the sign language encoder and the text encoder respectively for processing, until the processing of all predetermined layers is completed.

[0118] In step A2, the predetermined layer refers to the encoder layer that participates in cross-modal fusion. These layers achieve deep cross-modal semantic fusion by iteratively performing feature enhancement and propagation.

[0119] In this embodiment, enhanced visual features are input to the next layer of the sign language encoder, and enhanced text features are input to the next layer of the text encoder. The fusion process of steps 1032, 1033, and A1 is repeated in the last two layers until all six layers of encoding are completed. For example, after fusing to generate bidirectional enhanced features, which consist of enhanced visual features and enhanced text features, the bidirectional enhanced features are fed back to the original encoders of their respective modalities via residual connections as input to the next layer of encoding. The forward propagation process can be expressed by the following formula:

[0120] ;

[0121] in, Indicates the first The original characteristics of the layer; Indicates the source mode Enhanced current mode Features This indicates the corresponding current mode. The Layer encoder module; Indicates output up to the th Enhanced features of the layer.

[0122] In practical applications, for a 6-layer encoder, the last 2 layers participate in the fusion. Each layer performs bidirectional cross-modal attention interaction and feature enhancement, ultimately obtaining deeply fused visual and text features.

[0123] A3: Extract the first type of token from the enhanced visual features output from the last layer of the sign language encoder, and extract the second type of token from the enhanced text features output from the last layer of the text encoder.

[0124] In step A3, the first type of token refers to a special feature vector representing the global semantics of the entire sign language sequence, and the second type of token refers to a special feature vector representing the global semantics of the entire text sequence.

[0125] In this embodiment, the feature vector of the first position is extracted from the 128×128 enhanced visual features output from the 6th layer of the sign language encoder as the first type of token, and the feature vector of the first position is extracted from the enhanced text features output from the 6th layer of the text encoder as the second type of token.

[0126] In practical applications, the first type of token has a dimension of 128, and the second type of token has a dimension of 512, which encode the overall semantic information of the sign language sequence and the text sequence, respectively.

[0127] A4: Map the first type of token and the second type of token to the shared semantic space through their respective projection heads, and calculate the similarity of the corresponding features of the first type of token and the second type of token after mapping in the shared semantic space using dot product to obtain the global similarity.

[0128] In step A4, the projection head refers to a linear transformation layer that maps features of different dimensions to the same space; the shared semantic space refers to the vector space shared by visual and textual features; and global similarity refers to the degree of semantic matching between two modalities. The formula for calculating global similarity is: ;in This represents the global similarity between sign language and text. Indicates a global vision projection head. This represents the first type of token. Indicates the global projection header for text. This indicates the second type of token. This represents the transpose operator.

[0129] In this embodiment of the application, the 128-dimensional first-class token and the 512-dimensional second-class token are mapped to a 256-dimensional shared semantic space through independent linear layers, and then the dot product of the two mapped vectors is calculated as the global similarity.

[0130] In practical applications, the projection head achieves dimensionality transformation through a learnable weight matrix, and the global similarity calculated by the dot product is a scalar value that reflects the degree of semantic matching between the sign language sequence and the text sequence.

[0131] A5: Based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder, local similarity is generated.

[0132] Step A5 may specifically include the following steps:

[0133] Step B1: Input the enhanced text features output from the last layer of the sign language encoder and the enhanced visual features output from the last layer of the text encoder into a preset aggregator, extract multiple word tokens from the enhanced text features output from the last layer of the text encoder, and extract multiple visual feature tokens from the enhanced visual features output from the last layer of the sign language encoder.

[0134] In step B1, the aggregator is not a component in the early fusion network of sign language perception, but an independent functional module specifically designed to process text features to generate semantic clusters; word tokens refer to the feature vectors corresponding to each subword in the text sequence, and visual feature tokens refer to the visual feature vectors corresponding to each time step in the sign language sequence.

[0135] In this embodiment, the feature vectors of all positions in the enhanced text features output by the last layer of the text encoder are used as word tokens, and the feature vectors of all time steps in the enhanced visual features output by the last layer of the sign language encoder are used as visual feature tokens, which are then input into the aggregator for subsequent processing.

[0136] In practical applications, the text encoder outputs 16 word tokens, each with 512 dimensions; the sign language encoder outputs 128 visual feature tokens, each with 128 dimensions.

[0137] Step B2: Determine the clustering index for each word token.

[0138] In step B2, the cluster index refers to the semantic cluster number to which each text word belongs, and words in the same cluster are semantically coherent.

[0139] In this embodiment of the application, the semantic relationship between text sub-words is analyzed by the offset calculator in the aggregator, and then the clustering index of each text sub-word is dynamically determined by combining semantic coherence.

[0140] In practical applications, for a sequence containing 16 text sub-words, the sequence is divided into 8 semantic clusters by calculating the semantic similarity of adjacent sub-words. Each cluster contains 1-3 sub-words. For example, the sub-words "curios" and "ity" in "curiosity" are merged into 1 cluster.

[0141] Step B3: Based on the clustering index, merge the feature vectors of word tokens belonging to the same clustering result range to generate semantic clustering results.

[0142] In step B3, the feature vector of the word token is calculated through the forward propagation of the text encoder. It means that each text sub-word is a numerical representation with semantic information after being transformed by a multi-layer neural network. The semantic clustering result refers to the feature representation formed by merging multiple semantically related text sub-words.

[0143] In this embodiment of the application, word tokens belonging to the same cluster are merged by attention weighted pooling according to the clustering index to generate 8 semantic clustering results, each clustering result having a feature dimension of 512.

[0144] In practical applications, for a cluster containing 3 sub-words, the attention weights of each sub-word are calculated, and the 3 512-dimensional features are weighted and summed to obtain a 512-dimensional semantic clustering result.

[0145] Step B4: Calculate the cosine similarity between each visual feature token and all semantic clustering results, and select the maximum cosine similarity corresponding to each visual feature token.

[0146] In step B4, cosine similarity refers to the degree of similarity between two vectors in a direction, specifically the dot product of the two vectors divided by the product of their respective magnitudes.

[0147] In this embodiment of the application, for each of the 128 visual feature tokens, the cosine similarity between it and the 8 semantic clustering results is calculated, and then the maximum value among the 8 similarities is taken as the maximum cosine similarity corresponding to the visual token.

[0148] In practical applications, the cosine similarities between the first visual feature token and the eight clustering results are [0.85, 0.72, 0.63, 0.91, 0.78, 0.69, 0.82, 0.75], and the maximum value of 0.91 is taken as the maximum cosine similarity of the token.

[0149] Step B5: Calculate the weighted sum of the maximum cosine similarities corresponding to all visual feature tokens to obtain the local similarity.

[0150] In step B5, local similarity refers to the weighted sum of the maximum similarity between all visual tokens and the text cluster, reflecting the fine-grained degree of cross-modal matching.

[0151] In this embodiment, the maximum cosine similarity of the 128 visual feature tokens is weighted using softmax, and then weighted and summed to obtain the final local similarity. The formula for calculating this local similarity is as follows: ;in Indicates the local similarity between sign language and text. This indicates the total number of tokens of the first type. Indicates the first The characteristics of a first-class token Indicates the first The characteristics of a second-class token, where K is the total number of second-class tokens.

[0152] In practical applications, the 128 maximum cosine similarities are normalized by softmax to obtain a weight distribution, and the local similarity obtained by weighted summation is a scalar value between 0 and 1.

[0153] This application establishes a deep association between visual actions and linguistic semantics through multi-level cross-modal fusion and dual-granularity semantic alignment, which enhances the ability to distinguish fine-grained actions while maintaining overall semantic consistency.

[0154] Step 104: Based on the global similarity and the local similarity, determine the corresponding global contrast loss and local contrast loss, and coordinate the weights of the global contrast loss and the local contrast loss through a preset balance parameter to obtain the hierarchical loss.

[0155] In this embodiment, step 104 includes the following process:

[0156] Step 1041: Based on the global similarity, calculate the first global contrast loss from sign language to text and the second global contrast loss from text to sign language.

[0157] In step 1041, the first global contrastive loss refers to the contrastive learning loss with sign language features as query text features as keys, and the second global contrastive loss refers to the contrastive learning loss with text features as query sign language features as keys.

[0158] In this embodiment of the application, based on the global similarity matrix obtained in step 103, the contrast loss in two directions is calculated respectively. In the sign language to text direction, the similarity distribution between each sign language sample and all text samples is calculated, and in the text to sign language direction, the similarity distribution between each text sample and all sign language samples is calculated. The two contrast loss values ​​are obtained by softmax normalization and cross-entropy.

[0159] In practical applications, when the batch size is 16, the global similarity matrix is ​​16×16. The first global contrast loss calculates the matching degree between 16 sign language samples and 16 text samples, and the second global contrast loss calculates the matching degree between 16 text samples and 16 sign language samples. Both loss values ​​are between 0 and 10.

[0160] Step 1042: Based on the local similarity, calculate the first local contrast loss from sign language to text and the second local contrast loss from text to sign language.

[0161] In step 1042, the first local contrast loss refers to the fine-grained contrast loss from sign language to text calculated based on local similarity, and the second local contrast loss refers to the fine-grained contrast loss from text to sign language calculated based on local similarity.

[0162] In this embodiment of the application, based on the local similarity matrix obtained in step 103, fine-grained contrast loss is calculated in two directions respectively. The sign language to text direction focuses on the local matching of each visual token with the text cluster, and the text to sign language direction focuses on the local matching of each text cluster with the visual token. Two local contrast loss values ​​are calculated through contrastive learning.

[0163] In practical applications, the local similarity matrix is ​​16×16, the first local comparison loss value is 2.34, and the second local comparison loss value is 2.67. The two loss values ​​reflect the quality of fine-grained cross-modal matching.

[0164] Step 1043: Combine the first global contrast loss and the second global contrast loss into a global contrast loss, and combine the first local contrast loss and the second local contrast loss into a local contrast loss.

[0165] In step 1043, global contrast loss refers to the overall loss that integrates global contrast losses in two directions, and local contrast loss refers to the overall loss that integrates local contrast losses in two directions.

[0166] In this embodiment, the average of the first global contrast loss and the second global contrast loss is used to obtain the global contrast loss, and the average of the first local contrast loss and the second local contrast loss is used to obtain the local contrast loss.

[0167] In practical applications, the first global contrast loss is 3.21, the second global contrast loss is 3.45, and the average value is 3.33; the first local contrast loss is 2.34, the second local contrast loss is 2.67, and the average value is 2.505.

[0168] Step 1044: Assign weights to the global contrast loss and the local contrast loss using preset balance parameters.

[0169] In step 1044, the balancing parameter refers to the hyperparameter that controls the relative weights of the global contrast loss and the local contrast loss. It is used to adjust the importance of semantic alignment at different granularities and avoid feature bias.

[0170] In this embodiment, the balance parameter is set to 0.5. The global contrast loss is multiplied by 1 and subtracted from the balance parameter, and the local contrast loss is multiplied by the balance parameter to achieve the weight allocation of the two losses.

[0171] In practical applications, the balance parameter is set to 0.5. The global contrast loss of 3.33 multiplied by 0.5 is 1.665, and the local contrast loss of 2.505 multiplied by 0.5 is 1.2525.

[0172] Step 1045: Add the weighted global contrast loss to the weighted local contrast loss to obtain the hierarchical loss.

[0173] In step 1045, the hierarchical loss refers to the overall loss function that comprehensively considers global semantic alignment and local fine-grained alignment.

[0174] In this embodiment, the weighted global contrast loss and the weighted local contrast loss are directly added together to obtain the final hierarchical loss.

[0175] In practical applications, the weighted global contrast loss of 1.665 and the weighted local contrast loss of 1.2525 are added together to obtain a hierarchical loss of 2.9175.

[0176] This application achieves collaborative optimization of global semantics and local details through bidirectional contrastive learning and multi-granularity loss balancing, thereby improving the integrity and robustness of cross-modal semantic alignment.

[0177] Step 105: Based on the hierarchical loss, perform the sign language and text matching task and the language modeling task to obtain the matching loss and the language modeling loss respectively. The hierarchical loss, the matching loss and the language modeling loss are weighted and combined to obtain the pre-training total loss. Based on the pre-training total loss, the model parameters are adjusted collaboratively to complete the semantic enhancement pre-training process.

[0178] Among them, the sign language and text matching task refers to a binary classification task that determines whether the input sign language skeleton sequence and the corresponding text description are semantically matched. This task improves the model's ability to recognize the consistency between the two by strengthening the cross-modal alignment relationship between visual action features and language semantic features. The language modeling task refers to an autoregressive generation task that predicts one or more text tokens based on a given input text sequence. This task enhances the model's natural language understanding and generation capabilities by modeling the internal linguistic rules and semantic coherence of the text sequence.

[0179] In this embodiment, step 105 includes the following process:

[0180] Step 1051: Based on the hierarchical loss, perform a matching task in the target matching path, wherein the enhanced text features output from the last layer of the text encoder and the enhanced visual features output from the last layer of the sign language encoder are input into the multi-layer cross-attention module for deep interaction to generate matching probabilities, and the matching loss is calculated based on the matching probabilities.

[0181] In step 1051, the target matching path refers to a classification path specifically for determining whether a sign language sequence matches text, and it is generated by constructing a specialized multi-layer cross-attention network; the multi-layer cross-attention module refers to a deep network structure containing multiple cross-attention layers; the matching probability refers to the confidence score that the sign language sequence matches the text content; the matching loss refers to the difference between the predicted probability output by the target matching path and the true matching label, and this loss is calculated using a binary cross-entropy function to optimize the model's ability to determine whether the sign language sequence and text content match.

[0182] In this embodiment, the enhanced text features output from the last layer of the text encoder are used as the query, and the enhanced visual features output from the last layer of the sign language encoder are used as the key and value. These are input into a matching path containing four layers of cross-attention modules. After multiple layers of interaction, the matching probability is output through the classification head. Then, based on this probability and the true matching label, the binary cross-entropy loss is calculated as the matching loss.

[0183] In practical applications, the augmented text feature dimension is 16×512, the augmented visual feature dimension is 128×128, and after processing by 4 layers of cross-attention modules, the output matching probability is 0.87, the true label is 1, and the calculated matching loss is 0.24.

[0184] Step 1052: Simultaneously perform a language modeling task in the target modeling path, wherein the enhanced text features output from the last layer of the text encoder are input into a multi-layer self-attention module for autoregressive modeling to generate text token prediction results, and the language modeling loss is calculated based on the text token prediction results.

[0185] In step 1052, the target modeling path refers to the sequence generation path used for the language modeling task, which is generated by constructing a multi-layer self-attention network; the multi-layer self-attention module refers to a sequence modeling network containing multiple self-attention layers; the text token prediction result refers to the predicted distribution of the next token at each position in the text sequence; the language modeling loss refers to the difference measure between the text token prediction result output by the target modeling path and the real text token, which is calculated using the cross-entropy function and used to optimize the model's ability to generate text language and its semantic understanding ability.

[0186] In this embodiment, the enhanced text features output from the last layer of the text encoder are input into the target modeling path containing 6 layers of self-attention modules. Text tokens are predicted one by one through autoregression, and the cross-entropy loss between the prediction results and the real text tokens is calculated as the language modeling loss.

[0187] In practical applications, the enhanced text feature dimension is 16×512. After processing through 6 layers of self-attention modules, the output text token prediction result is 2.15 compared with the language modeling loss calculated from the real text.

[0188] Step 1053: The hierarchical loss, the matching loss, and the language modeling loss are linearly combined according to the preset first weight coefficient, second weight coefficient, and third weight coefficient to obtain the pre-training total loss.

[0189] In step 1053, the first weight coefficient refers to the weight of the hierarchical loss, the second weight coefficient refers to the weight of the matching loss, and the third weight coefficient refers to the weight of the language modeling loss. The pre-training total loss refers to the overall optimization target obtained by weighting and summing the hierarchical loss, matching loss, and language modeling loss according to the preset weight coefficients. This loss comprehensively reflects the overall performance of the model in multiple tasks such as cross-modal alignment, matching judgment, and language generation.

[0190] In this embodiment, the first weight coefficient is set to 0.4, the second weight coefficient is set to 0.3, and the third weight coefficient is set to 0.3. The hierarchical loss is multiplied by the first weight coefficient, the matching loss is multiplied by the second weight coefficient, and the language modeling loss is multiplied by the third weight coefficient. Then, the three weighted losses are added together to obtain the total pre-training loss.

[0191] In practical applications, the hierarchical loss is 2.92, the matching loss is 0.24, and the language modeling loss is 2.15. The weighted calculation process is 2.92×0.4+0.24×0.3+2.15×0.3=1.168+0.072+0.645=1.885, resulting in a total pre-training loss of 1.885.

[0192] Step 1054: Based on the pre-trained total loss, simultaneously adjust all trainable parameters in the sign language perception early fusion network, the target matching path, and the target modeling path.

[0193] In step 1054, trainable parameters refer to the weights and biases in the model that are optimized by the gradient descent algorithm. These trainable parameters include the encoder parameters in the early fusion network for sign language perception, the cross-attention layer parameters in the matching path, and the self-attention layer parameters in the target modeling path.

[0194] In this embodiment, the AdamW optimizer can be used to calculate the gradient based on the pre-trained total loss, and the parameters of the sign language encoder and text encoder in the early fusion network of sign language perception, the parameters of the 4-layer cross-attention module in the target matching path, and the parameters of the 6-layer self-attention module in the target modeling path can be updated simultaneously through the backpropagation algorithm.

[0195] In practical applications, the pre-training total loss of 1.885 is used to calculate the gradients of each parameter through backpropagation. All trainable parameters are updated using a learning rate of 0.0001. These trainable parameters include 1.28 million parameters of the sign language encoder, 230 million parameters of the text encoder, 4.2 million parameters of the matching path, and 6.8 million parameters of the target modeling path.

[0196] Step 1055: Perform multiple rounds of iterative updates on all trainable parameters until the total pre-training loss reaches a preset convergence threshold.

[0197] In step 1055, the convergence threshold refers to the critical value at which the total pre-training loss stops decreasing, and is used to determine whether the model training is complete. In this embodiment of the application, the value of the preset convergence threshold is not specifically limited, and can be set according to the actual situation.

[0198] In this embodiment of the application, the convergence threshold is set to 0.1. During the training process, the change in the total pre-training loss is continuously monitored. Training is stopped when the loss decrease is less than the convergence threshold for 10 consecutive training cycles.

[0199] In practical applications, after 25 rounds of training, the total pre-training loss of the model decreased from the initial 3.56 to 0.08, and the loss decreased by less than 0.1 for 10 consecutive rounds, reaching the convergence condition and completing the pre-training process for semantic enhancement.

[0200] This application achieves deep enhancement of cross-modal semantic representation through multi-task collaborative learning and adaptive parameter optimization, providing a high-quality pre-trained model foundation for downstream sign language understanding tasks.

[0201] Step 106: In the fine-tuning stage, based on the adjusted model parameters and in combination with the target task type, fine-tune some parameters of the semantic enhancement pre-trained sign language understanding framework to achieve enhanced understanding of sign language semantics.

[0202] In step 106, the target task type includes three specific tasks: isolated sign language recognition, continuous sign language recognition, and sign language translation without Gloss; fine-tuning refers to the process of optimizing some parameters based on pre-trained model parameters using specific task data; some parameters include the top-level parameters of the encoder and all parameters of the decoder, wherein the bottom-level parameters of the encoder remain unchanged in the pre-training state.

[0203] In this embodiment, based on the optimized parameters obtained during the pre-training stage, the corresponding output layer structure is configured according to the specific target task type. For isolated sign language recognition tasks, the classification output layer is used to predict single words; for continuous sign language recognition tasks, the sequence output layer is used to predict word sequences; and for Gloss-free sign language translation tasks, the language generation layer is used to predict natural sentences. Then, supervised fine-tuning of the encoder's top-level parameters and decoder parameters is performed using specific task data, while keeping the encoder's bottom-level parameters unchanged. This allows the model to adapt to the specific needs of the task while maintaining general semantic understanding capabilities.

[0204] In other embodiments, during the pre-training phase, the embodiments of this application can also perform unified verification on English sign language (such as How2Sign, OpenASL) and Chinese sign language (such as CSL-Daily) datasets based on the above method, thereby proving that the pre-trained knowledge is transferable.

[0205] Figure 3 A schematic diagram of the structure of a skeleton-based semantic enhancement pre-trained sign language understanding framework provided in this application embodiment is shown below. Figure 3 As shown, the detailed implementation section describes:

[0206] The acquisition module 31 is used to acquire sign language video data, skeletal sequences, and text data paired with the skeletal sequences.

[0207] Extraction module 32 is used to extract skeletal key point sequences from the sign language video data, model the skeletal key point sequences using a spatial-temporal graph convolutional network to form skeletal features, and perform word segmentation on the text data.

[0208] Input module 33 is used to input the skeletal features and segmented text data into the early fusion network for sign language perception during the pre-training stage. Through the cross-attention mechanism, it generates visual features for text guidance and text features for visual guidance, and performs two-level semantic alignment on the visual features and text features to determine global similarity and local similarity.

[0209] The determination module 34 is used to determine the corresponding global contrast loss and local contrast loss based on the global similarity and the local similarity, and to coordinate the weights of the global contrast loss and the local contrast loss through a preset balance parameter to obtain the hierarchical loss.

[0210] The combination module 35 is used to perform a sign language and text matching task and a language modeling task based on the hierarchical loss, obtain the matching loss and the language modeling loss respectively, and weight the hierarchical loss, the matching loss and the language modeling loss to obtain the pre-training total loss. Based on the pre-training total loss, the model parameters are coordinated to complete the semantic enhancement pre-training process.

[0211] The fine-tuning module 36 is used to fine-tune some parameters of the semantic enhancement pre-trained sign language understanding framework based on the adjusted model parameters and in combination with the target task type during the fine-tuning stage, so as to achieve enhanced understanding of sign language semantics.

[0212] The system of the skeleton-based semantic enhancement pre-training sign language understanding framework in this application embodiment is used to implement the aforementioned method of the skeleton-based semantic enhancement pre-training sign language understanding framework. Therefore, the specific implementation of the system of the skeleton-based semantic enhancement pre-training sign language understanding framework can be found in the embodiment section of the method of the skeleton-based semantic enhancement pre-training sign language understanding framework above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0213] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described methods for a skeleton-based semantic enhancement pre-trained sign language understanding framework.

[0214] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described methods for a skeleton-based semantic enhancement pre-trained sign language understanding framework.

[0215] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0216] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for a skeleton-based semantic enhancement pre-trained sign language understanding framework.

[0217] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0218] The above provides a detailed description of the method and system for a skeleton-based semantic enhancement pre-training sign language understanding framework provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for pre-training a sign language understanding framework based on skeleton semantic enhancement, characterized in that, include: Acquire sign language video data, skeletal sequences, and text data paired with the skeletal sequences; The skeletal key point sequence is extracted from the sign language video data, and the skeletal key point sequence is modeled using a spatial-temporal graph convolutional network to form skeletal features. The text data is then segmented into words. During the pre-training phase, the skeletal features and segmented text data are input into the early fusion network for sign language perception. A cross-attention mechanism is used to generate visual features and text features for text guidance. The visual and text features are then subjected to two-level semantic alignment to determine global and local similarities. The early fusion network for sign language perception includes a multi-layered sign language encoder and a multi-layered text encoder. A first type of token is extracted from the enhanced visual features output from the last layer of the sign language encoder, and a second type of token is extracted from the enhanced text features output from the last layer of the text encoder. The first and second types of tokens are mapped to a shared semantic space using corresponding projection heads. The similarity of the corresponding features after mapping in the shared semantic space is calculated using dot product to obtain the global similarity. Local similarities are generated based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder. Based on the global similarity and the local similarity, the corresponding global contrast loss and local contrast loss are determined. The weights of the global contrast loss and the local contrast loss are coordinated by a preset balance parameter to obtain the hierarchical loss. Based on the hierarchical loss, a sign language and text matching task and a language modeling task are performed to obtain the matching loss and the language modeling loss respectively. The hierarchical loss, the matching loss and the language modeling loss are weighted and combined to obtain the total pre-training loss. Based on the total pre-training loss, the model parameters are adjusted collaboratively to complete the semantic enhancement pre-training process. During the fine-tuning phase, based on the adjusted model parameters and in conjunction with the target task type, some parameters of the semantic enhancement pre-trained sign language understanding framework are fine-tuned to achieve enhanced understanding of sign language semantics. The generation of local similarity based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder includes: The enhanced text features output from the last layer of the sign language encoder and the enhanced visual features output from the last layer of the text encoder are input into a preset aggregator. Multiple word tokens are extracted from the enhanced text features output from the last layer of the text encoder, and multiple visual feature tokens are extracted from the enhanced visual features output from the last layer of the sign language encoder. Determine the clustering index for each word token; Based on the clustering index, the feature vectors of word tokens belonging to the same clustering result range are merged to generate semantic clustering results; Calculate the cosine similarity between each visual feature token and all semantic clustering results, and select the maximum cosine similarity corresponding to each visual feature token; The local similarity is obtained by weighted summation of the maximum cosine similarity corresponding to all visual feature tokens.

2. The method according to claim 1, characterized in that, The process involves inputting the skeletal features and segmented text data into a sign language perception early fusion network. Through a cross-attention mechanism, visual features for text guidance and text features for visual guidance are generated. A two-level semantic alignment is then performed on the visual features and text features to determine global and local similarity, including: In each predetermined layer of the sign language encoder, a cross-attention mechanism is used to calculate the first attention weight from text to vision, and visual features for text guidance are generated based on the first attention weight. In each predetermined layer of the text encoder, a cross-attention mechanism is used to calculate the second attention weight from vision to text, and visually guided text features are generated based on the second attention weight. Global semantic alignment and local semantic alignment are performed on the visual features and the text features respectively to generate corresponding global similarity and local similarity.

3. The method according to claim 2, characterized in that, The step of performing global semantic alignment and local semantic alignment on the visual features and the text features respectively includes: The visual features are added to the original visual features of the current layer to obtain enhanced visual features, and the text features are added to the original text features of the current layer to obtain enhanced text features. The enhanced visual features and the enhanced text features are respectively input into the next layer of the sign language encoder and the text encoder for processing, until the processing of all predetermined layers is completed.

4. The method according to claim 1, characterized in that, The step of extracting skeletal keypoint sequences from the sign language video data and modeling the skeletal keypoint sequences using a spatial-temporal graph convolutional network to form skeletal features includes: Obtain the skeletal key point sequence of the target area from the sign language video data; For the skeletal keypoint sequence of each part, a space-time graph convolutional network is used to model the joint relationships in the spatial dimension and the motion trajectory in the temporal dimension, respectively, to obtain the spatiotemporal dynamic features of each part. The spatiotemporal dynamic features of each part are mapped to a unified dimensional space through linear transformation to generate corresponding compact features; Connect the compact features of all parts along the feature dimension to form skeletal features.

5. The method according to claim 1, characterized in that, Based on the hierarchical loss, a sign language and text matching task and a language modeling task are performed to obtain the matching loss and the language modeling loss, respectively. The hierarchical loss, the matching loss, and the language modeling loss are then weighted and combined to obtain the pre-training total loss. Based on the pre-training total loss, the model parameters are collaboratively adjusted, including: Based on the hierarchical loss, a matching task is performed in the target matching path, wherein the enhanced text features output from the last layer of the text encoder and the enhanced visual features output from the last layer of the sign language encoder are input into the multi-layer cross-attention module for deep interaction to generate matching probabilities, and the matching loss is calculated based on the matching probabilities. Simultaneously, a language modeling task is performed in the target modeling path, wherein the enhanced text features output from the last layer of the text encoder are input into a multi-layer self-attention module for autoregressive modeling to generate text token prediction results, and the language modeling loss is calculated based on the text token prediction results. The hierarchical loss, the matching loss, and the language modeling loss are linearly combined according to the preset first weight coefficient, second weight coefficient, and third weight coefficient to obtain the total pre-training loss; Based on the pre-trained total loss, all trainable parameters in the sign language perception early fusion network, the target matching path, and the target modeling path are simultaneously adjusted. The trainable parameters are iteratively updated multiple times until the total pre-training loss reaches a preset convergence threshold.

6. The method according to claim 1, characterized in that, Based on the global similarity and the local similarity, the corresponding global contrast loss and local contrast loss are determined. The weights of the global contrast loss and the local contrast loss are coordinated using a preset balance parameter to obtain the hierarchical loss, including: Based on the global similarity, the first global contrast loss from sign language to text and the second global contrast loss from text to sign language are calculated respectively. Based on the local similarity, the first local contrast loss from sign language to text and the second local contrast loss from text to sign language are calculated respectively. The first global contrast loss and the second global contrast loss are combined into a global contrast loss, and the first local contrast loss and the second local contrast loss are combined into a local contrast loss. The global contrast loss and the local contrast loss are weighted according to preset balance parameters. The weighted global contrast loss is added to the weighted local contrast loss to obtain the hierarchical loss.

7. A system for pre-training a skeleton-based semantic enhancement sign language understanding framework, characterized in that, include: The acquisition module is used to acquire sign language video data, skeletal sequences, and text data paired with the skeletal sequences; The extraction module is used to extract the skeletal key point sequence from the sign language video data, model the skeletal key point sequence using a spatial-temporal graph convolutional network to form skeletal features, and perform word segmentation on the text data. The input module, during the pre-training phase, inputs the skeletal features and segmented text data into the early fusion network for sign language perception. Through a cross-attention mechanism, it generates text-guided visual features and visual-guided text features, and performs two-level semantic alignment on the visual and text features to determine global and local similarities. The early fusion network for sign language perception includes a multi-layered sign language encoder and a multi-layered text encoder. It extracts a first type of token from the enhanced visual features output from the last layer of the sign language encoder and a second type of token from the enhanced text features output from the last layer of the text encoder. The first and second types of tokens are mapped to a shared semantic space using corresponding projection heads. The similarity of the corresponding features after mapping the first and second types of tokens in the shared semantic space is calculated using dot product to obtain the global similarity. Local similarities are generated based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder. The determination module is used to determine the corresponding global contrast loss and local contrast loss based on the global similarity and the local similarity, and to coordinate the weights of the global contrast loss and the local contrast loss through a preset balance parameter to obtain the hierarchical loss; The combination module is used to perform sign language and text matching tasks and language modeling tasks based on the hierarchical loss, obtain matching loss and language modeling loss respectively, and weight and combine the hierarchical loss, the matching loss and the language modeling loss to obtain the pre-training total loss. Based on the pre-training total loss, the model parameters are coordinated to complete the semantic enhancement pre-training process. The fine-tuning module is used in the fine-tuning stage to fine-tune some parameters of the semantic enhancement pre-trained sign language understanding framework based on the adjusted model parameters and in combination with the target task type, so as to achieve enhanced understanding of sign language semantics. The generation of local similarity based on the enhanced visual features output from the last layer of the sign language encoder and the enhanced text features output from the last layer of the text encoder includes: The enhanced text features output from the last layer of the sign language encoder and the enhanced visual features output from the last layer of the text encoder are input into a preset aggregator. Multiple word tokens are extracted from the enhanced text features output from the last layer of the text encoder, and multiple visual feature tokens are extracted from the enhanced visual features output from the last layer of the sign language encoder. Determine the clustering index for each word token; Based on the clustering index, the feature vectors of word tokens belonging to the same clustering result range are merged to generate semantic clustering results; Calculate the cosine similarity between each visual feature token and all semantic clustering results, and select the maximum cosine similarity corresponding to each visual feature token; The local similarity is obtained by weighted summation of the maximum cosine similarity corresponding to all visual feature tokens.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for a skeleton-based semantic enhancement pre-trained sign language understanding framework as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the method for a skeleton-based semantic enhancement pre-trained sign language understanding framework as described in any one of claims 1 to 6.

Images