Sign language recognition methods, devices and equipment

Abstract

This invention provides a sign language recognition method, apparatus, and device, relating to the field of image recognition technology. The method includes: acquiring a sequence of image frames to be recognized, and determining skeletal detection results based on the image frame sequence; inputting the image frame sequence and skeletal detection results into a pre-trained sign language recognition model to obtain the sign language recognition result corresponding to the image frame sequence; wherein the sign language recognition model includes a cross-modal attention fusion module; the cross-modal attention fusion module is used to calculate a first semantic enhancement result of the visual feature vector on the skeletal feature vector, and a second semantic enhancement result of the skeletal feature vector on the visual feature vector, and based on the first and second semantic enhancement results, determining a frame-level prediction probability sequence; the frame-level prediction probability sequence is used to determine the sign language recognition result. This invention can fully leverage the advantages of multimodal feature complementarity, effectively improving the accuracy of sign language recognition.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a sign language recognition method, apparatus, and device. Background Technology

[0002] Continuous sign language recognition systems primarily rely on deep learning models to model the spatiotemporal features of video frame sequences when performing sign language recognition tasks. Existing methods typically employ convolutional neural networks (CNNs), 3D convolutional networks (3D CNNs), or hybrid structures combining recurrent neural networks (RNNs, LSTMs) to extract spatial features and temporal variation information from video frame sequences, achieving the conversion from continuous sign language videos to word sequences. Currently, most sign language recognition systems are trained based on single-modal visual information, meaning they only use video frame sequences as input. Such methods can achieve good recognition results under standardized shooting conditions, but when complex backgrounds, uneven lighting, action occlusion, or changes in perspective appear in the video, the stability of the model's feature extraction significantly decreases, affecting recognition accuracy.

[0003] To further improve the accuracy of sign language recognition, related technologies have begun to incorporate skeletal data, optical flow information, or depth images to form multimodal sign language recognition frameworks. These frameworks can, to some extent, compensate for the shortcomings of single-modal visual features. However, our research has found that existing multimodal sign language recognition frameworks generally employ static feature stitching or linear weighted fusion for multimodal feature fusion. Since different modal data differ in distribution, temporal alignment, and information representation, the fusion effect of these static feature stitching or linear weighted fusion mechanisms is limited. This fails to fully leverage the complementary advantages of multimodal features, making it difficult to capture subtle hand movements and structural changes, thus hindering the effective improvement of sign language recognition accuracy. Summary of the Invention

[0004] This invention provides a sign language recognition method, apparatus, and device to address the problem that existing sign language recognition models' multimodal feature fusion mechanisms cannot fully leverage the complementary advantages of multimodal features, making it difficult to capture subtle hand movements and structural changes, thus hindering the effective improvement of sign language recognition accuracy.

[0005] In a first aspect, embodiments of the present invention provide a sign language recognition method, including: Obtain the image frame sequence to be identified, and determine the skeleton detection result based on the image frame sequence; The image frame sequence and the skeleton detection results are input into a pre-trained sign language recognition model to obtain the sign language recognition results corresponding to the image frame sequence. The sign language recognition model includes a cross-modal attention fusion module; The cross-modal attention fusion module is used to calculate a first semantic enhancement result of the visual feature vector on the skeletal feature vector, and a second semantic enhancement result of the skeletal feature vector on the visual feature vector, and to determine a frame-level prediction probability sequence based on the first semantic enhancement result and the second semantic enhancement result; the frame-level prediction probability sequence is used to determine the sign language recognition result; The visual feature vector is based on the image frame sequence. Secondly, embodiments of the present invention provide a sign language recognition device, comprising: An acquisition module is used to acquire a sequence of image frames to be identified and to determine the skeleton detection result based on the sequence of image frames; The recognition module is used to input the image frame sequence and the skeleton detection result into a pre-trained sign language recognition model to obtain the sign language recognition result corresponding to the image frame sequence; The sign language recognition model includes a cross-modal attention fusion module; The cross-modal attention fusion module is used to calculate a first semantic enhancement result of the visual feature vector on the skeletal feature vector, and a second semantic enhancement result of the skeletal feature vector on the visual feature vector, and to determine a frame-level prediction probability sequence based on the first semantic enhancement result and the second semantic enhancement result; the frame-level prediction probability sequence is used to determine the sign language recognition result; The visual feature vector is obtained by feature extraction based on the image frame sequence; the skeletal feature vector is obtained by feature extraction based on the skeletal detection results.

[0006] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0007] In this embodiment of the invention, the first semantic enhancement result reflects the semantic alignment attention of the visual feature vector to the skeletal feature vector, and the second semantic enhancement result reflects the semantic guidance of the skeletal feature vector to the visual feature vector. By calculating the semantic alignment attention of the visual feature vector to the skeletal feature vector (i.e., the first semantic enhancement result) and the semantic guidance of the skeletal feature vector to the visual feature vector (i.e., the second semantic enhancement result), the sign language recognition model can make the visual feature vector and the skeletal feature vector mutually guide each other, thereby achieving dynamic interaction between them. This enhances the temporal consistency and complementarity between the visual and skeletal feature vectors, thereby accurately determining the frame-level prediction probability sequence and ultimately improving the accuracy of sign language recognition results. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating the implementation of the sign language recognition method provided in this embodiment of the invention; Figure 2 These are key point diagrams provided in the embodiments of the present invention; Figure 3 This is a skeleton structure diagram provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the skeletal angle provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the sign language recognition model provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the S3D network in the visual branch provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the cross-modal attention fusion module provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the probability distribution sequence corresponding to LIEB sign language lexical units provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the sign language recognition device provided in an embodiment of the present invention. Detailed Implementation

[0009] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0010] Multimodal sign language recognition frameworks can, to some extent, compensate for the shortcomings of single-modal visual features. However, our research has found that existing multimodal sign language recognition frameworks generally employ static feature splicing or linear weighted fusion for multimodal feature fusion. Since different modal data differ in distribution, temporal alignment, and information representation, the fusion effect of these static feature splicing or linear weighted fusion mechanisms is limited. This fails to fully leverage the complementary advantages of multimodal features, making it difficult to capture subtle hand movements and structural changes, thus hindering the effective improvement of sign language recognition accuracy.

[0011] To fully leverage the advantages of multimodal feature complementarity and thus improve sign language recognition accuracy, the sign language recognition model in this embodiment of the invention includes a cross-modal attention fusion module. This module calculates the first semantic enhancement result of the visual feature vector on the skeletal feature vector, and the second semantic enhancement result of the skeletal feature vector on the visual feature vector. This allows the visual feature vector and the skeletal feature vector to guide each other, thereby achieving dynamic interaction between them and enhancing their complementarity. This, in turn, accurately determines the frame-level prediction probability sequence, ultimately improving the accuracy of sign language recognition results.

[0012] See Figure 1 The flowchart illustrating the implementation of the sign language recognition method provided in this embodiment of the invention is described in detail below: Step 101: Obtain the image frame sequence to be identified, and determine the skeleton detection result based on the image frame sequence.

[0013] This invention provides an embodiment that can acquire video clips containing sign language gestures and obtain a sequence of image frames to be recognized by sampling these video clips. This image frame sequence contains multiple consecutive frames of images to be recognized. This embodiment of the invention can then input this image frame sequence into a subsequent sign language recognition model for sign language recognition.

[0014] In sign language recognition, relying solely on image frame sequences is insufficient to resist noise factors such as lighting variations, background interference, and occlusion. This is especially true when processing gestures with small amplitude or localized blur, such as visually similar signs (VISigns), where significant performance bottlenecks exist. To combat these noise factors and improve sign language recognition accuracy, this invention extracts the corresponding skeleton detection results from the image frame sequence and inputs both the skeleton detection results and the image frame sequence into the sign language recognition model. This enhances the model's ability to model structural posture changes and its robustness, ultimately improving sign language recognition accuracy.

[0015] The specific implementation method for determining bone detection results based on image frame sequences is as follows: The above image frame sequence is input into a pre-trained HRNet (High-Resolution Network) model, and the HRNet model outputs the keypoint detection results corresponding to the image frame sequence. Here, the HRNet model can be trained using the COCO-WholeBody dataset to obtain a pre-trained HRNet model.

[0016] The pre-trained HRNet model estimates human keypoints for each frame in the image frame sequence, obtaining the two-dimensional spatial coordinates of a total of 79 keypoints covering areas such as hands, arms, shoulders, and face. Here, each keypoint is a skeleton point.

[0017] For example, the image frame sequence in this embodiment of the invention may contain 64 frames of images. Correspondingly, the keypoint detection results output by the HRNet model can be represented as a three-dimensional tensor. .

[0018] The first dimension is the time frame, representing 64 frames of images. The second dimension is the keypoint number, representing 79 keypoints. The third dimension is the two-dimensional spatial coordinates, representing the two-dimensional spatial coordinates of each keypoint.

[0019] Considering that simple joint position features cannot fully depict the movement patterns and structural changes in sign language, this invention, in order to more effectively capture the dynamic relationships and posture information between bones, further constructs bone vectors and bone angles based on the aforementioned keypoint detection results. Here, the keypoint detection results, bone vectors, and bone angles together constitute the aforementioned bone detection results.

[0020] For each frame in the above image frame sequence, the HRNet model can determine the corresponding keypoint map for that frame. For example, see... Figure 2 The keypoint map contains all the keypoints in the frame image. In this embodiment of the invention, all bone edges in each keypoint map can be determined based on the predefined bone edge structure in the skeleton structure, thereby determining the skeleton structure map.

[0021] Here, for each keypoint map, the bone edges in the keypoint map can be determined based on the following formula (1): (1) in, Indicate key points and key points The skeletal structure formed by it and Indicate key points Two-dimensional spatial coordinates, and Indicate key points The two-dimensional spatial coordinates.

[0022] For each keypoint map, all bone edges can be determined based on the above formula (1) to obtain the corresponding skeleton structure map. For example, the skeleton structure map can be as follows: Figure 3 As shown.

[0023] All bone edges in all image frames can form a bone vector. .in, This indicates the number of predefined bone edges in the skeleton structure. For example, You can use 62 to cover the common skeletal borders of the upper body and hands.

[0024] To further model the curvature and pose changes between keypoints, this embodiment of the invention can use each keypoint with two bone edges in the skeleton structure diagram as the angle center point. For each angle center point, the included angle between the two bone edges corresponding to that angle center point is calculated. The formula for calculating the included angle can be expressed as: (2) in, Indicates the center point of the angle The included angle between the corresponding two skeletal sides, Indicates the center point of the angle and key points The skeletal structure formed by it Indicate key points and key The skeletal structure formed by it.

[0025] The aforementioned angles can reflect the degree of angular curvature between adjacent bone edges, effectively revealing the movement trends of the elbow, wrist, and other parts during continuous movements.

[0026] For each skeleton structure diagram, all the included angles can be determined based on the above formula (2) to obtain a schematic diagram of the bone included angles corresponding to that key point diagram. For example, the schematic diagram of the bone included angles can be as shown in 4.

[0027] All the included angles in all the skeletal structure diagrams can constitute skeletal angles. .

[0028] Step 102: Input the image frame sequence and skeleton detection results into the pre-trained sign language recognition model to obtain the sign language recognition results corresponding to the image frame sequence.

[0029] In this embodiment of the invention, the aforementioned image frame sequence, key point detection results, bone vectors, and bone angles can all be input into a pre-trained sign language recognition model to obtain the final sign language recognition result. Here, the sign language recognition result can be represented in the form of a sign language lexical sequence, which contains at least one sign language lexical to reflect the semantic recognition result corresponding to the sign language gesture in the image frame sequence.

[0030] In some embodiments, the sign language recognition model may include a cross-modal attention fusion (CMAF) module.

[0031] The cross-modal attention fusion module is used to calculate the first semantic enhancement result of the visual feature vector on the skeletal feature vector, and the second semantic enhancement result of the skeletal feature vector on the visual feature vector, and to determine the frame-level prediction probability sequence based on the first and second semantic enhancement results.

[0032] Here, the cross-modal attention fusion module constructs mutually guiding bidirectional attention paths, which can calculate the first semantic enhancement result of visual feature vectors on skeletal feature vectors and the second semantic enhancement result of skeletal feature vectors on visual feature vectors, thereby modeling the semantic relationship between image frames and skeletons, and the semantic relationship between skeletons and image frames, and enhancing the complementarity between visual feature vectors and skeletal feature vectors.

[0033] Based on the first and second semantic enhancement results obtained above, this embodiment of the invention enhances the skeletal feature vector and the visual feature vector using the first and second semantic enhancement results respectively, and then fuses the enhanced skeletal feature vector and the visual feature vector to obtain a fused feature vector. Based on the fused feature vector, a frame-level prediction probability sequence is determined.

[0034] Here, the frame-level prediction probability sequence includes: the prediction sign language lexicon probability sequence corresponding to each frame image; the prediction sign language lexicon probability sequence includes the prediction probability value corresponding to different sign language lexicons.

[0035] In this embodiment of the invention, K different sign language lexical units can be preset. The cross-modal attention fusion module can determine the predicted probability values ​​of the K sign language lexical units corresponding to each frame image based on the corresponding fused feature vectors. Here, K can be an integer greater than 1. This embodiment of the invention obtains the final sign language recognition result by decoding and recognizing the frame-level predicted probability sequence.

[0036] In some embodiments, see Figure 5 The sign language recognition model may include a visual branch 51, a skeletal branch 52, a cross-modal attention fusion module 53, and a decoding recognition module 54.

[0037] Visual branch 51 can extract features from the image frame sequence to obtain visual feature vectors. Skeletal branch 52 can extract features from the above skeleton detection results (including keypoint detection results, skeleton vectors, and skeleton angles) to obtain skeleton feature vectors.

[0038] The cross-modal attention fusion module 53 is used to calculate the first semantic enhancement result and the second semantic enhancement result, and to determine the frame-level prediction probability sequence based on the first semantic enhancement result and the second semantic enhancement result.

[0039] The decoding and recognition module 54 is used to decode and recognize the above frame-level prediction probability sequence to obtain the final sign language recognition result.

[0040] The sign language recognition models will be introduced in detail below.

[0041] The vision branch primarily performs efficient and detailed spatiotemporal feature extraction on image frame sequences. Considering the numerous dynamic changes and spatial displacements in sign language gestures during recognition, especially visually similar but semantically different gestures (VISigns), the sign language recognition model needs strong local spatiotemporal modeling capabilities to avoid confusion. Therefore, this invention introduces an improved Separable 3D Convolution (S3D) network as the backbone structure of the vision branch, aiming to improve the model's perception capabilities of dynamic regions of the hands, face, and body while controlling computational costs.

[0042] Specifically, see Figure 6 The visual branch employs an S3D network consisting of four convolutional blocks as its backbone. Within the S3D network, each convolutional block comprises spatial convolution (3×3) and temporal convolution (kernel size 3, stride 1), performing frame-by-frame spatial feature modeling and inter-frame dynamic information modeling, respectively. The spatial and temporal convolution operations are followed by BatchNorm and ReLU activation functions to enhance the model's nonlinear modeling capability and data stability.

[0043] To further enhance the model's ability to perceive key regions (hands and face), see [link / reference]. Figure 6 In this embodiment of the invention, a squeeze-and-excitation (SE) module is introduced at the output of the BatchNorm and ReLU activation functions following the temporal convolution to explicitly model channel importance. The SE module performs global average pooling on each feature channel, and then learns the channel attention weight vector through two fully connected layers (with a compression ratio of 16) and an activation function. Ultimately, the response to semantically salient regions is enhanced through a weighted approach. Here, the data processing procedure of the SE module can be expressed using the following formula: (3) (4) in, This represents the input data for the SE module. This indicates a global average pooling operation. This represents the weight matrix of the first fully connected layer in the SE module. Represents the ReLU activation function. This represents the weight matrix of the second fully connected layer in the SE module. This represents the Sigmoid activation function. Represents the channel attention weight vector. This indicates the output data of the SE module.

[0044] The SE module mentioned above is used to enhance the model's ability to comprehensively model high-frequency dynamic features (such as finger movements) and low-frequency posture changes (such as arm trajectories).

[0045] The output of the SE module is followed by a max pooling operation (kernel size 2×2, step size 2) to achieve spatial downsampling. All pooling operations only apply to the spatial dimension (H / W), while the time dimension (T) remains at 64.

[0046] For each frame of the input vision branch, after four convolutional blocks and four spatial downsampling operations, the spatial size of the input image is reduced from 224×224 to 14×14, while the temporal dimension remains unchanged at 64. The shape of the output vision tensor at this point is... .in 832 represents the batch size and the number of channels.

[0047] To accommodate the subsequent cross-modal attention fusion module, the vision branch performs global average pooling on the output tensor in the spatial dimension and downsamples it once in the temporal dimension with a stride of 4, resulting in an output shape of... A three-dimensional vector. Wherein, This represents the time dimension after time downsampling. , That is, the visual feature vector ultimately output by the visual branch. .

[0048] The skeletal branch mainly includes three feature extraction branches and a feature summing module. The first feature extraction branch has the same structure as the vision branch, including an S3D backbone network composed of four convolutional blocks, and global average pooling and temporal downsampling operations are set at the output of the S3D backbone network.

[0049] The key point detection results from the skeleton detection can be fed into the first feature extraction branch (also known as the skeleton S3D network) in the form of a heatmap to extract the main skeleton features. .

[0050] The second and third feature extraction branches have the same structure, both including a fully connected layer. The bone vector and bone angle are mapped to the same dimension through the fully connected layer, resulting in... .

[0051] The feature summation module is used to perform frame-level summation of the outputs of each feature extraction branch to obtain the final skeletal feature vector: (5) in, Represents the skeletal feature vector. This indicates the main skeletal flow characteristics described above. These represent the bone angle and bone vector after mapping via the fully connected layer, respectively.

[0052] The aforementioned skeletal feature vectors, as the final representation of the skeletal branch, are fed together with the visual feature vectors output by the visual branch into the cross-modal attention fusion module.

[0053] To achieve modal alignment and time synchronization between skeletal and visual modalities, this embodiment of the invention provides a unified frame number normalization and skeleton interpolation mechanism: First, each image frame sequence is uniformly resampled to 64 frames. Samples with more than 64 frames are downsampled at equal intervals; samples with fewer than 64 frames are expanded by mirroring. This strategy ensures strict alignment between the visual and skeletal modalities at the frame level. Second, the extraction of skeletal points (i.e., keypoints) is performed frame-by-frame based on the image frame sequence. However, due to pose occlusion, recognition failures, etc., keypoints may be missing. Here, linear interpolation methods can be used to fill in the missing coordinates on the time axis, ensuring that each frame has a complete skeleton representation. This ensures a one-to-one correspondence between the final skeletal sequence and the image sequence at the frame level.

[0054] This invention constructs a two-stream, three-dimensional separable convolutional network (i.e., a visual branch and a skeletal branch with an S3D network as the backbone), used to process image frame sequences and skeletal detection results, respectively. The visual branch is responsible for extracting appearance features such as color, texture, and hand contour; the skeletal branch is used to model structural features such as hand joint positions, motion trajectories, and angle changes. The two branches operate independently and in parallel, and finally, a feature mapping layer unifies the feature dimensions, achieving modal synchronization and providing high-quality input for subsequent cross-modal fusion. This structure effectively avoids the sensitivity of single-modal models to issues such as illumination and occlusion, improving feature diversity and information completeness.

[0055] Based on the determination of skeletal feature vectors and visual feature vectors, this embodiment of the invention utilizes a cross-modal attention fusion module to perform cross-modal fusion of skeletal feature vectors and visual feature vectors to obtain a frame-level prediction probability sequence.

[0056] The decoding and recognition module obtains the final sign language recognition result by decoding and recognizing the frame-level predicted probability sequence. Here, the decoding and recognition module can be a Connectionist Temporal Classification (CTC) decoder.

[0057] The CTC decoder can identify the sign language word with the highest predicted probability value for each frame image from the frame-level prediction probability sequence, and designate this sign language word as the sign language word corresponding to that frame image. Then, by merging consecutive repeated words and removing whitespace, the redundant frame-level classification results are compressed into a concise word sequence, resulting in the final sign language recognition result.

[0058] In this embodiment of the invention, the first semantic enhancement result reflects the semantic alignment attention of the visual feature vector to the skeletal feature vector, and the second semantic enhancement result reflects the semantic guidance of the skeletal feature vector to the visual feature vector. By calculating the semantic alignment attention of the visual feature vector to the skeletal feature vector (i.e., the first semantic enhancement result) and the semantic guidance of the skeletal feature vector to the visual feature vector (i.e., the second semantic enhancement result), the sign language recognition model can make the visual feature vector and the skeletal feature vector mutually guide each other, thereby achieving dynamic interaction between them. This enhances the temporal consistency and complementarity between the visual and skeletal feature vectors, thereby accurately determining the frame-level prediction probability sequence and ultimately improving the accuracy of sign language recognition results.

[0059] The cross-modal attention fusion module will be described in detail below.

[0060] In some embodiments, see Figure 7 The cross-modal attention fusion module includes: a convolutional unit 71, a bidirectional attention mechanism unit 72, a residual connection mechanism unit 73, a fusion unit 74, and a transformation unit 75.

[0061] Convolutional unit 71 is used to perform convolution processing on visual feature vectors and skeletal feature vectors to obtain visual convolutional features and skeletal convolutional features.

[0062] Here, the convolutional unit 71 contains two convolutional blocks, each of which performs convolution processing on the visual feature vector and the skeletal feature vector respectively to obtain visual convolutional features and skeletal convolutional features.

[0063] Bidirectional attention mechanism unit 72, used for: Using visual convolutional features as queries and skeletal convolutional features as keys and values, we calculate the first semantic enhancement result of the visual feature vector to the skeletal feature vector. Using skeletal convolutional features as queries and visual convolutional features as keys and values, we calculate the second semantic enhancement result of the skeletal feature vector for the visual feature vector. Residual connection mechanism unit 73, used for: The bone feature vector is augmented using the first semantic enhancement result to obtain bone enhancement features; The visual feature vector is augmented using the second semantic enhancement result to obtain visually enhanced features. The fusion unit 74 is used to concatenate the skeletal enhancement features and visual enhancement features, and to perform linear transformation and nonlinear activation on the concatenated features to obtain a fused feature vector. Transformation unit 75 is used to perform linear transformation and nonlinear normalization on the fused feature vector to obtain a frame-level prediction probability sequence.

[0064] Specifically, the bidirectional attention mechanism unit 72 can calculate the first semantic enhancement result and the second semantic enhancement result according to the following formula (6): (6) in, This represents the result of the first semantic enhancement. This indicates the result of the second semantic enhancement. This represents the softmax activation function. Representing visual convolutional features, Represents skeletal convolution features. , , , , and Represents the learnable projection matrix. Indicates transpose. Indicates the dimension of the projection matrix.

[0065] Here, the bidirectional attention mechanism unit 72 can explicitly model the semantic dependencies between vision and skeleton at each frame time step (i.e., each frame image), enhancing semantic guidance and modality perception capabilities. The second semantic enhancement result output by the bidirectional attention mechanism unit 72 is shown below. This is the semantic enhancement result of the skeletal feature vector on the visual feature vector; the first semantic enhancement result. It is the semantic enhancement result of visual feature vectors on skeletal features.

[0066] To preserve the representational power of the modality itself during modality fusion and prevent overfitting or attention drift, this embodiment of the invention introduces a residual connection mechanism unit. This unit sums the original modality features (i.e., skeletal feature vectors and visual feature vectors) with the corresponding semantic enhancement results frame by frame to construct residual enhancement representations (i.e., skeletal enhancement features and visual enhancement features). The specific calculation formula is as follows: (7) in, Indicates visual enhancement features, This indicates bone enhancement features.

[0067] The aforementioned residual enhancement design helps to mitigate information loss and semantic drift during deep fusion and provides the model with a more stable feature propagation path.

[0068] Next, to fully integrate the complementary information of the two modalities, this embodiment of the invention utilizes a fusion unit to concatenate the aforementioned visual enhancement features and skeletal enhancement features along the channel dimension, and employs a fully connected layer for dimensionality compression and semantic fusion. Here, the fully connected layer is primarily used to perform linear transformations and nonlinear activations. The data processing flow of the fusion unit can be expressed using the following calculation formula: (8) (9) in, This represents the stitched feature resulting from the splicing of visual enhancement features and skeletal enhancement features along the channel dimension. This indicates a channel splicing operation. Represents the fused feature vector. This represents a nonlinear activation function, used to introduce nonlinear expressive power. This represents the linear fusion weight matrix. , This represents the bias vector. .

[0069] Fusion feature vectors This characterizes the spatiotemporal semantic representation of the fused two modalities in each frame. For example, embodiments of the present invention can unify the dimension of the fused feature vector to... .

[0070] Transformation unit 75 may contain a fully connected layer and a Softmax function to perform linear transformation and nonlinear normalization on the fused feature vector to obtain a frame-level prediction probability sequence. This frame-level prediction probability sequence is the input feature sequence of the CTC decoder.

[0071] This bidirectional attention mechanism can model the semantic associations and dependencies between features of two modalities, thereby dynamically and purposefully fusing key information. The resulting fused feature vector contains richer and more discriminative cross-modal semantic information, providing higher-quality input for subsequent decoding and recognition, and helping to improve the accuracy of the final sign language recognition results.

[0072] This invention constructs a bidirectional attention path that guides each other, and models the semantic relationship between vision and skeleton and the semantic relationship between skeleton and vision respectively. The first semantic enhancement result and the second semantic enhancement result can be calculated respectively, realizing the dynamic interaction modeling and temporal consistency enhancement between visual features and skeleton features. On this basis, the residual information of the two modalities are fused to construct a joint feature flow (i.e., fused feature vector) with stronger semantic representation ability, thereby enhancing the complementarity between features and the ability to distinguish complex gestures (especially VISigns).

[0073] This invention introduces a cross-modal attention fusion module, enabling dynamic weight allocation of visual and skeletal features at the frame level. Through a query, key, and value mechanism, visual and skeletal features serve as mutual attention sources, achieving adaptive alignment and interactive updates between modalities. This mechanism allows the sign language recognition model to automatically determine the primary information source based on scene changes: enhancing skeletal flow weights when visual information is disturbed and strengthening visual flow features when bones are missing, thereby maintaining overall recognition stability. This fusion strategy significantly improves feature utilization and recognition robustness compared to static stitching methods, making it particularly suitable for sign language recognition tasks involving continuous actions and complex environments.

[0074] In sign language recognition tasks, gloss refers to the symbolic representation of each independent word in a sign language video, i.e., sign language lexical units, which constitute the output label sequence of the sign language recognition model. The gloss sequence preserves the temporal order of sign language but removes grammatical variations, serving as an important intermediate form connecting video frame sequences and linguistic symbols. For example, in the Phoenix-2014 dataset, the entire sign language lexical table consists of 1295 glosses, each participating in model classification training as an independent category. Because the visual modal expression of sign language recognition is highly subjective, and there is a high degree of visual similarity between sign language lexical units (e.g., “NIEDERSCHLÄGE” (precipitation) / “SCHNEE” (snowfall)), the model is highly susceptible to confusion when relying solely on appearance features. Traditional one-hot labeling treats each gloss as an independent discrete category, failing to reflect their semantic relationship. When two glosses are visually similar but semantically different (e.g., VISigns), the sign language recognition model is prone to confusion based solely on appearance features. To address the aforementioned issues, this invention introduces a language-aware label smoothing loss function to train the sign language recognition model, thereby mitigating the ambiguity risk caused by visual homogeneity and improving the model's ability to distinguish VISigns.

[0075] In some embodiments, the training process of a sign language recognition model may include the following steps: Step A: Obtain a pre-defined sign language lexicon table and construct a semantic embedding space based on the sign language lexicon table. Here, the sign language lexicon table contains multiple sign language lexicons. The semantic embedding space contains word vectors corresponding to each sign language lexicon.

[0076] For example, see Figure 5 Each sign language lexical can be input into a pre-trained FastText model, and the FastText model will output the corresponding word vector for that sign language lexical. Here, the dimension of the word vector corresponding to each sign language lexical can be 300. This word vector consists of 300 lexical units that are semantically the same as or similar to that sign language lexical.

[0077] Here, the FastText model can be trained on a large corpus (e.g., Common Crawl), which can capture the co-occurrence rules of word meaning in natural language and is the foundation for constructing a semantic space.

[0078] Here, the pre-defined sign language dimension table can be represented as follows: .in, , and These represent each sign language lexical in the sign language lexical table. Each sign language lexical corresponds to a word vector with a dimension of 300. Here, the sign language lexical table can include K sign language lexicals.

[0079] Step B involves calculating the semantic similarity between the word vector corresponding to each sign language lexicon in the sign language lexicon table and the word vectors corresponding to the remaining sign language lexicon, and determining the probability distribution sequence corresponding to the sign language lexicon based on the semantic similarity.

[0080] Here, the semantic similarity can be calculated using the following formula (10): (13) in, Indicates sign language words and sign language words Semantic similarity between them Indicates sign language words The corresponding word vectors, Indicates sign language words The corresponding word vectors.

[0081] For each sign language word, embodiments of the present invention can calculate the semantic similarity between that sign language word and other sign language words. Furthermore, it is also necessary to determine the semantic similarity between that sign language word and itself. By normalizing all semantic similarities (including its own semantic similarity) corresponding to that sign language word, a probability distribution sequence corresponding to that sign language word is obtained, which can also be called a soft-label distribution.

[0082] For each target lexical (i.e., any sign language lexical in the sign language lexical table), the length of the probability distribution sequence of the target lexical is equal to the number of sign language lexicals K in the sign language lexical table. Each element in the probability distribution sequence corresponds to the probability value of a sign language lexical in the sign language lexical table, which is used to reflect the semantic relevance or similarity between the target lexical and that sign language lexical.

[0083] For example, see Figure 8 The probability distribution sequence corresponding to the sign language word LIEB is: [0.7, 0.1, 0.1, 0.05, 0.025, 0.025]. That is, for the target word LIEB, it has a similarity of 0.7 to itself (LIEB), meaning there is a 0.7 probability that it is itself; a similarity of 0.14 to Liebe, meaning there is a 0.14 probability that it is Liebe; a similarity of 0.1 to lieben, meaning there is a 0.1 probability that it is lieben; a similarity of 0.05 to Freund, meaning there is a 0.05 probability that it is Freund; a similarity of 0.025 to nett, meaning there is a 0.025 probability that it is nett; and a similarity of 0.025 to other sign language words, meaning there is a 0.025 probability that it is another sign language word.

[0084] This invention, by determining the probability distribution sequence corresponding to each sign language lexical, retains the dominant weight of the real sign language lexical labels while giving other semantically similar categories a certain activation probability. This guides the model to focus on the semantic relationship between categories during training, alleviates the confusion problem of visually indistinguishable lexicals, reduces the ambiguity risk caused by visual homogeneity, and improves the model's ability to distinguish VISigns.

[0085] To balance the weight differences between the main tag (i.e., the target word) and synonym tags, this embodiment of the invention introduces a smoothing coefficient to control the smoothness of the probability distribution. Based on the introduction of the smoothing coefficient, the probability distribution sequence corresponding to each sign language word can be represented as: (14) in, Indicates the first Sign language morpheme The corresponding probability distribution sequence, Indicates the first Sign language morpheme The probability value corresponding to the first sign language word in the corresponding probability distribution sequence. Indicates the first Sign language morpheme The corresponding probability distribution sequence of the th The probability value corresponding to each sign language word. Indicates the first Sign language morpheme The corresponding probability distribution sequence of the th The probability value corresponding to each sign language word. Indicates the total number of sign language lexical units. Indicates the first Sign language morpheme The corresponding word vector and the first Sign language morpheme The semantic similarity between corresponding word vectors This represents the smoothing coefficient.

[0086] Here, when When the probability approaches 1, the above probability distribution sequence degenerates into a one-hot label; when As the probability increases, the probability distribution tends to smooth out, and the sign language recognition model pays more attention to semantic structure. The value of can be obtained by tuning parameters on the validation set to balance the main label weight and semantic nearest neighbor representation. For example, .

[0087] Step C: Based on the probability distribution sequence corresponding to each sign language word, construct a language perception label smoothing loss function.

[0088] In this embodiment of the invention, the probability distribution sequence corresponding to each sign language word can be used as an auxiliary supervision signal to participate in model training in order to solve the problem of word confusion.

[0089] Specifically, a language-aware label smoothing loss function can be constructed based on the probability distribution sequence corresponding to each sign language word and the frame-level predicted probability sequence output by the cross-modal attention fusion module in the sign language recognition model. This loss function not only encourages the sign language recognition model to output a predicted distribution that conforms to the target label, but also utilizes the structural information of the semantic space to prevent the sign language recognition model from being overconfident in misclassification, especially demonstrating stronger discriminative ability in highly confusing scenarios.

[0090] In some embodiments, the specific implementation of constructing the language-aware label smoothing loss function is as follows: Step A1: For the training samples in the training process of the sign language recognition model, the frame-level prediction probability sequence determined by the cross-modal attention fusion module can be obtained.

[0091] Here, the frame-level prediction probability sequence contains the prediction sign language lexical probability sequence corresponding to each frame of the training samples. Specifically, the prediction sign language lexical probability sequence contains the prediction probability values ​​corresponding to different sign language lexical terms.

[0092] For a single training sample, for each frame of the image in that training sample, the cross-modal attention fusion module can output a predicted sign language lexical probability sequence corresponding to that frame. The length of this predicted sign language lexical probability sequence is K (i.e., the number of sign language lexical units in the sign language lexical unit table). This predicted sign language lexical probability sequence contains the predicted probability values ​​corresponding to K sign language lexical units (i.e., each sign language lexical unit in the sign language lexical unit table).

[0093] Step B1: Obtain the real sign language lexical units corresponding to each frame of the training sample, and determine the real sign language lexical unit probability sequence based on the real sign language lexical units and the probability distribution sequence corresponding to each sign language lexical unit.

[0094] For a training sample, for each frame of the training sample, the embodiments of the present invention can obtain the real sign language words corresponding to the frame image and determine the probability distribution sequence corresponding to the real sign language words as the probability sequence of the real sign language words corresponding to the frame image.

[0095] Step C1: Based on the real sign language lexical probability sequence and the predicted sign language lexical probability sequence corresponding to each frame image, construct a language-aware label smoothing loss function.

[0096] The embodiments of the present invention can calculate the cross-entropy loss between the real sign language lexical probability sequence and the predicted sign language lexical probability sequence, thereby constructing a language perception label smoothing loss function.

[0097] In some embodiments, the language-aware label smoothing loss function can be expressed as the following equation (15): (15) in, This represents the language-aware label smoothing loss function. Indicates the total number of sign language lexical units. Indicates the first The probability sequence of real sign language lexical units corresponding to the frame image. This represents the probability sequence of predicted sign language morphemes corresponding to the current frame image. The predicted probability value corresponding to each sign language word.

[0098] It should be noted that, to improve the model's generalization ability during training, this embodiment of the invention only introduces the aforementioned probability distribution sequence during the training phase; the argmax inference strategy is still used during testing, thus avoiding introducing computational burden during the deployment phase. This loss function is decoupled from the main recognition task and seamlessly integrated into the overall model training process as a lightweight auxiliary path.

[0099] A pre-trained word vector model (FastText) is used to calculate the semantic similarity between sign language words, constructing a semantic soft-label matrix. During model training, a semantic relevance distribution is used instead of traditional one-hot labels, guiding the network to learn the distance relationships between categories in the semantic space. When visually similar gestures (such as similar hand shapes but different meanings) appear, the model can adjust the classification boundary based on semantic distance, effectively distinguishing between synonymous or easily confused gestures. This strategy significantly reduces the misrecognition rate of VISigns-class gestures, improving the model's semantic recognition ability and generalization performance.

[0100] Step D: Train the sign language recognition model based on the language-aware label smoothing loss function to obtain the trained sign language recognition model.

[0101] Considering that existing technologies for training sign language recognition models primarily rely on CTC loss for sequence alignment optimization, lacking multi-level supervision signals, and that relying solely on a single loss function can easily lead to local optima in the early stages of training, resulting in unstable learning processes and impacting final recognition accuracy, this invention, when training the sign language recognition model using the language-aware label smoothing loss function, introduces a primary classification loss function and a visual branch-assisted classification loss function. This constructs a multi-objective combined loss function to balance sequence-level alignment, semantic discrimination, and visual feature discrimination capabilities, achieving end-to-end optimized training of the sign language recognition system.

[0102] Specifically, for the training samples in the training process of the sign language recognition model, the sign language recognition results output by the sign language recognition model and the visual output tensor determined by the visual branch in the sign language recognition model are obtained. Then, based on the sign language recognition results, a main classification loss function is constructed. Subsequently, based on the visual output tensor, the visual feature recognition results are determined, and based on the visual feature recognition results, a visual branch-assisted classification loss function is constructed. Next, the main classification loss function, the visual branch-assisted classification loss function, and the language perception label smoothing loss function are weighted and fused to obtain the total loss function. Finally, the sign language recognition model is trained using the total loss function to obtain the trained sign language recognition model.

[0103] Here, the main classification loss function (also known as the CTC loss function) serves as a supervisory signal for the main recognition path, addressing the issues of inconsistent lengths and lack of alignment annotations between the input frame sequence and the output word sequence. This loss function encourages the network output to achieve the highest confidence on implicitly aligned paths by maximizing the sum of probabilities across all possible paths. The main classification loss function can be expressed as: (16) in, Represents the primary classification function. Indicates the first Alignment path, This represents the real sign language lexical sequence corresponding to the training samples. Represents all possible sequences that map to real sign language lexical units. The set of alignment paths, This represents a frame-level prediction probability sequence. The sign language recognition model is indicated by the first The probability of the alignment path outputting the sign language recognition result is the core component of CTC loss.

[0104] To enhance the capabilities of the visual branch in discriminative local action modeling, particularly its representation and discrimination of dynamic hand regions, see [reference needed]. Figure 5 In this embodiment of the invention, a separate auxiliary classification path is designed for the visual branch. This path is independent of the main recognition network and is supervised by single-modal prediction based on visual features, focusing on improving the model's sensitivity to low-frequency, highly confusing classes. The auxiliary path uses fully connected layers and a softmax function to perform linear transformation and nonlinear normalization on the visual output tensor output in the visual branch, thereby obtaining the predicted sign language lexical probability sequence corresponding to each frame of the image, i.e., the visual feature recognition result.

[0105] Considering the problem of extremely imbalanced word distribution, and to avoid the sign language recognition model being overly biased towards high-frequency categories, this embodiment of the invention introduces a category reweighting mechanism. Specifically, the word frequency of each category (i.e., each sign language word) in the training set is statistically analyzed, and weights are constructed using the logarithmic inverse function: (17) in, Indicates the first Word frequency of each sign language word element This is used to effectively suppress the dominant weight of high-frequency categories on the loss, while increasing the gradient contribution of low-frequency categories during training.

[0106] Based on formula (17), the embodiments of the present invention can construct a visual branch-assisted classification loss function: (18) in, This represents the visual branch-assisted classification loss function. Indicates batch size, This represents the number of image frames in the training samples. Indicates the total number of sign language lexical units. Indicates the first The weight of each sign language word, Indicates the number of the current batch In the training samples, the th The probability sequence of real sign language lexical units corresponding to the frame image The probability value corresponding to each sign language word. Indicates the number of the current batch In the visual feature recognition results of the training sample, the first... The first frame of the predicted sign language lexical probability sequence corresponding to the image The predicted probability value corresponding to each sign language word.

[0107] Ultimately, the total loss function consists of three terms, jointly optimizing the main and auxiliary paths of the model: (19) For example, it can be set ,in, The optimal balance can be achieved in the parameter tuning of the validation set.

[0108] In this embodiment of the invention, when training the model using training samples, the total loss function can be calculated based on the model prediction results output by the training samples and the corresponding real results. If the total loss function does not converge, the model parameters can be adjusted and the loss function can be recalculated until the loss function converges, thus obtaining a trained sign language recognition model.

[0109] In the first 20 rounds of model training, the first two convolutional blocks in the visual branch can be frozen, and only the last two convolutional blocks and the model parameters of other modules can be fine-tuned to stabilize the distribution of the underlying general representation. After the cross-modal attention fusion module has basically converged, all four convolutional blocks are unfrozen for joint optimization, thereby achieving semantic adaptation and modal alignment.

[0110] This invention ensures consistency between the recognized sequence and the temporal order through a primary classification loss function (CTC loss function). Building upon this, a language-aware label smoothing loss function is combined to enhance semantic discriminability, and a visual branch-assisted classification loss function is introduced to strengthen local frame-level supervision. Multi-task loss joint training optimizes the model simultaneously in temporal modeling, semantic discrimination, and feature learning, accelerating model convergence and improving overall recognition accuracy.

[0111] This invention integrates spatiotemporal feature fusion and semantic perception within the same training framework, achieving a three-level joint optimization of "vision-structure-semantics". The model exhibits excellent performance in semantic consistency and temporal continuity, accurately recognizing complete hand gestures composed of multiple actions.

[0112] The sign language recognition model in this embodiment of the invention comprehensively optimizes the sign language recognition process from three dimensions: feature layer, structural layer and semantic layer, which significantly improves the model's recognition accuracy, robustness and semantic understanding ability.

[0113] First, at the feature level, this invention employs a dual-stream S3D spatiotemporal coding structure (i.e., a visual branch and a skeletal branch composed of an S3D backbone network). The RGB video stream (i.e., image frame sequences) and the skeletal stream (skeleton detection results) are input into independent 3D convolutional networks for feature extraction. The RGB stream captures hand texture, motion details, and spatial variations, while the skeletal stream models structural information such as joint positions, bone vectors, and angles. This independent modeling of dual-stream features allows the model to simultaneously obtain complementary representations of appearance and structural information, significantly improving its feature representation capabilities in complex motion and occluded scenes. Experimental results show that this structure reduces the word error rate (WER) by approximately 4.5% on the PHOENIX-2014T dataset and improves the accuracy by approximately 6.2% on the CSL-Daily dataset compared to the single-modal CNN-LSTM model.

[0114] Secondly, at the structural level, a cross-modal attention mechanism is introduced to achieve dynamic fusion of visual feature vectors and skeletal feature vectors. Through bidirectional attention calculation, the sign language recognition model can automatically allocate modal weights at each time step based on the current action features, enabling adaptive alignment and complementarity of information between different modalities in the temporal dimension. This mechanism effectively solves the problems of modal imbalance and temporal mismatch in traditional static fusion methods. When there are changes in lighting or background interference in the video, the model can automatically enhance the skeletal feature weights, thereby maintaining recognition stability. Experimental results show that after adopting cross-modal attention, the model's recognition accuracy in complex background videos is improved by approximately 8.3%, and its robustness in scenarios with partial hand occlusion is significantly enhanced.

[0115] Furthermore, at the semantic level, a language-aware label smoothing loss function is proposed. This function incorporates a word vector model (FastText) to calculate the semantic similarity between sign language words, generating a semantic soft-label distribution (i.e., a probability distribution sequence) to replace the traditional one-hot label training method. During model training, this loss function guides the model to learn the distance relationships between words in the semantic space, enabling the model to distinguish visually similar but semantically different gestures, thus significantly reducing the confusion rate of visually similar gestures (VISigns). Comparative experiments show that after adding the language-aware label smoothing loss function, the model's recognition accuracy on a subset of similar gestures improves by approximately 9.7%, and the global WER metric decreases by an average of approximately 2.1%.

[0116] Finally, at the training and optimization level, this invention simultaneously constrains the model's temporal alignment capability, semantic recognition capability, and feature discrimination capability through a joint loss function, forming a multi-objective collaborative optimization mechanism. This joint strategy not only accelerates the model's convergence speed but also reduces overfitting, significantly improving the model's transfer performance across different datasets.

[0117] This invention achieves high-precision recognition of complex action sequences in continuous sign language videos through the synergistic effect of three key technologies: dual-stream feature encoding (i.e., visual and skeletal branches), cross-modal dynamic fusion (i.e., cross-modal attention fusion module), and semantic perception optimization (i.e., language perception label smoothing loss function). Compared with existing methods, this invention demonstrates higher recognition accuracy and stronger generalization ability on publicly available datasets. It also achieves significant improvements in handling issues such as lighting changes, action occlusion, and confusion of similar gestures, effectively supporting the application of sign language recognition systems in scenarios such as practical communication, assisted education, and communication with the hearing impaired.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0119] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0120] Figure 9 A schematic diagram of the sign language recognition device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 9 As shown, the sign language recognition device 9 includes an acquisition module 91 and a recognition module 92.

[0121] The acquisition module 91 is used to acquire the image frame sequence to be identified and determine the skeleton detection result based on the image frame sequence; The recognition module 92 is used to input the image frame sequence and skeleton detection results into the pre-trained sign language recognition model to obtain the sign language recognition results corresponding to the image frame sequence; The sign language recognition model includes a cross-modal attention fusion module. The cross-modal attention fusion module is used to calculate the first semantic enhancement result of the visual feature vector on the skeletal feature vector, and the second semantic enhancement result of the skeletal feature vector on the visual feature vector, and to determine the frame-level prediction probability sequence based on the first and second semantic enhancement results; the frame-level prediction probability sequence is used to determine the sign language recognition result. The visual feature vector is obtained by feature extraction based on the image frame sequence; the skeletal feature vector is obtained by feature extraction based on the skeletal detection results.

[0122] In one possible implementation, the cross-modal attention fusion module includes: a convolutional unit, a bidirectional attention mechanism unit, a residual connection mechanism unit, a fusion unit, and a transformation unit; Convolutional units are used to perform convolution processing on visual feature vectors and skeletal feature vectors to obtain visual convolutional features and skeletal convolutional features. Bidirectional attention mechanism unit, used for: Using visual convolutional features as queries and skeletal convolutional features as keys and values, we calculate the first semantic enhancement result of the visual feature vector to the skeletal feature vector. Using skeletal convolutional features as queries and visual convolutional features as keys and values, we calculate the second semantic enhancement result of the skeletal feature vector for the visual feature vector. The residual connection mechanism unit is used for: The bone feature vector is augmented using the first semantic enhancement result to obtain bone enhancement features; The visual feature vector is augmented using the second semantic enhancement result to obtain visually enhanced features. The fusion unit is used to concatenate the skeletal enhancement features and visual enhancement features, and to perform linear transformation and nonlinear activation on the concatenated features to obtain the fused feature vector; The transformation unit is used to perform linear transformation and nonlinear normalization on the fused feature vector to obtain a frame-level prediction probability sequence.

[0123] In one possible implementation, the bidirectional attention mechanism unit is used for: according to The first semantic enhancement result and the second semantic enhancement result are calculated; in, This represents the result of the first semantic enhancement. This indicates the result of the second semantic enhancement. This represents the softmax activation function. Representing visual convolutional features, Represents skeletal convolution features. , , , , and Represents the learnable projection matrix. Indicates transpose. Indicates the dimension of the projection matrix.

[0124] In one possible implementation, the identification module 92 is further used for: Obtain a pre-defined sign language lexicon table and construct a semantic embedding space based on the sign language lexicon table; the sign language lexicon table contains multiple sign language lexicons; the semantic embedding space contains word vectors corresponding to each sign language lexicon. For each sign language lexicon in the sign language lexicon table, calculate the word vector corresponding to the sign language lexicon and the semantic similarity between it and the word vectors corresponding to the remaining sign language lexicon. Then, determine the probability distribution sequence corresponding to the sign language lexicon based on the semantic similarity. Based on the probability distribution sequence corresponding to each sign language word, a language perception label smoothing loss function is constructed. The sign language recognition model is trained using the language-aware label smoothing loss function to obtain a well-trained sign language recognition model.

[0125] In one possible implementation, the identification module 92 is specifically used for: For each sign language lexicon in the sign language lexicon table, according to Determine the probability distribution sequence corresponding to the sign language lexical; in, Indicates the first Sign language morpheme The corresponding probability distribution sequence, Indicates the first Sign language morpheme The probability value corresponding to the first sign language word in the corresponding probability distribution sequence. Indicates the first Sign language morpheme The corresponding probability distribution sequence of the th The probability value corresponding to each sign language word. Indicates the first Sign language morpheme The corresponding probability distribution sequence of the th The probability value corresponding to each sign language word. Indicates the total number of sign language lexical units. Indicates the first Sign language morpheme The corresponding word vector and the first Sign language morpheme The semantic similarity between corresponding word vectors This represents the smoothing coefficient.

[0126] In one possible implementation, the identification module 92 is specifically used for: For the training samples in the training process of the sign language recognition model, the frame-level prediction probability sequence determined by the cross-modal attention fusion module is obtained; the frame-level prediction probability sequence contains the prediction sign language word probability sequence corresponding to each frame of the training sample; the prediction sign language word probability sequence contains the prediction probability value corresponding to different sign language words. Obtain the real sign language words corresponding to each frame of the training samples, and determine the probability sequence of real sign language words based on the real sign language words and the probability distribution sequence of each sign language word. Based on the real sign language lexical probability sequence and the predicted sign language lexical probability sequence corresponding to each frame of image, a language-aware label smoothing loss function is constructed.

[0127] In one possible implementation, the identification module 92 is specifically used for: according to Construct a language-aware label smoothing loss function; in, This represents the language-aware label smoothing loss function. Indicates the total number of sign language lexical units. Indicates the first The probability sequence of real sign language lexical units corresponding to the frame image. This represents the probability sequence of predicted sign language morphemes corresponding to the current frame image. The predicted probability value corresponding to each sign language word.

[0128] In one possible implementation, the identification module 92 is specifically used for: For the training samples in the training process of the sign language recognition model, obtain the sign language recognition results output by the sign language recognition model and the visual output tensor determined by the visual branch in the sign language recognition model. Based on the sign language recognition results, a main classification loss function is constructed; Based on the visual output tensor, the visual feature recognition result is determined, and based on the visual feature recognition result, a visual branch-assisted classification loss function is constructed. The main classification loss function, the visual branch auxiliary classification loss function, and the language perception label smoothing loss function are weighted and fused to obtain the total loss function; The sign language recognition model is trained using the total loss function to obtain a well-trained sign language recognition model.

[0129] This device embodiment can be used to implement the above method embodiment, and its technical principle and implementation effect are the same as those of the above method embodiment, so they will not be repeated here.

[0130] This invention also provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the above method embodiments.

[0131] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0132] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A sign language recognition method, characterized in that, include: Obtain the image frame sequence to be identified, and determine the skeleton detection result based on the image frame sequence; The image frame sequence and the skeleton detection results are input into a pre-trained sign language recognition model to obtain the sign language recognition results corresponding to the image frame sequence. The sign language recognition model includes a cross-modal attention fusion module; The cross-modal attention fusion module is used to calculate a first semantic enhancement result of the visual feature vector on the skeletal feature vector, and a second semantic enhancement result of the skeletal feature vector on the visual feature vector, and to determine a frame-level prediction probability sequence based on the first semantic enhancement result and the second semantic enhancement result; the frame-level prediction probability sequence is used to determine the sign language recognition result; The visual feature vector is obtained by feature extraction based on the image frame sequence; the skeletal feature vector is obtained by feature extraction based on the skeletal detection results.

2. The sign language recognition method according to claim 1, characterized in that, The cross-modal attention fusion module includes: a convolutional unit, a bidirectional attention mechanism unit, a residual connection mechanism unit, a fusion unit, and a transformation unit; The convolutional unit is used to perform convolution processing on the visual feature vector and the skeletal feature vector to obtain visual convolutional features and skeletal convolutional features. The bidirectional attention mechanism unit is used for: Using the visual convolutional features as the query and the skeletal convolutional features as the key and value, calculate the first semantic enhancement result of the visual feature vector for the skeletal feature vector; Using the skeletal convolutional features as the query and the visual convolutional features as the key and value, the second semantic enhancement result of the skeletal feature vector for the visual feature vector is calculated; The residual connection mechanism unit is used for: The bone feature vector is subjected to residual enhancement using the first semantic enhancement result to obtain bone enhancement features; The visual feature vector is subjected to residual enhancement using the second semantic enhancement result to obtain visually enhanced features; The fusion unit is used to concatenate the skeletal enhancement features and the visual enhancement features, and to perform linear transformation and nonlinear activation on the concatenated features to obtain a fused feature vector. The transformation unit is used to perform linear transformation and nonlinear normalization on the fused feature vector to obtain the frame-level prediction probability sequence.

3. The sign language recognition method according to claim 2, characterized in that, The bidirectional attention mechanism unit is used for: according to The first semantic enhancement result and the second semantic enhancement result are calculated. in, This represents the first semantic enhancement result. This represents the result of the second semantic enhancement. This represents the softmax activation function. This represents the visual convolutional features. This represents the skeletal convolution feature. , , , , and Represents the learnable projection matrix. Indicates transpose. Indicates the dimension of the projection matrix.

4. The sign language recognition method according to any one of claims 1-3, characterized in that, Before inputting the image frame sequence and the skeleton detection results into the pre-trained sign language recognition model, the method further includes: A preset sign language lexicon table is obtained, and a semantic embedding space is constructed based on the sign language lexicon table; the sign language lexicon table contains multiple sign language lexicons; the semantic embedding space contains word vectors corresponding to each sign language lexicon. For each sign language lexicon in the sign language lexicon table, calculate the word vector corresponding to the sign language lexicon and the semantic similarity between it and the word vectors corresponding to the remaining sign language lexicon, and determine the probability distribution sequence corresponding to the sign language lexicon based on the semantic similarity. Based on the probability distribution sequence corresponding to each sign language word, a language perception label smoothing loss function is constructed. The sign language recognition model is trained based on the language-aware label smoothing loss function to obtain a trained sign language recognition model.

5. The sign language recognition method according to claim 4, characterized in that, For each sign language lexical in the aforementioned sign language lexical table, calculate the semantic similarity between the word vector corresponding to that sign language lexical and the word vectors corresponding to the remaining sign language lexicals, and determine the probability distribution sequence corresponding to that sign language lexical based on each semantic similarity, including: For each sign language lexicon in the aforementioned sign language lexicon table, according to Determine the probability distribution sequence corresponding to the sign language lexical; in, Indicates the first Sign language morpheme The corresponding probability distribution sequence, Indicates the first Sign language morpheme The probability value corresponding to the first sign language word in the corresponding probability distribution sequence. Indicates the first Sign language morpheme The corresponding probability distribution sequence of the th The probability value corresponding to each sign language word. Indicates the first Sign language morpheme The corresponding probability distribution sequence of the th The probability value corresponding to each sign language word. Indicates the total number of sign language lexical units. Indicates the first Sign language morpheme The corresponding word vector and the first Sign language morpheme The semantic similarity between corresponding word vectors This represents the smoothing coefficient.

6. The sign language recognition method according to claim 4, characterized in that, The language-aware label smoothing loss function is constructed based on the probability distribution sequence corresponding to each sign language lexical unit, including: For the training samples in the training process of the sign language recognition model, a frame-level prediction probability sequence is obtained by the cross-modal attention fusion module; the frame-level prediction probability sequence includes the prediction sign language word probability sequence corresponding to each frame of the training sample; the prediction sign language word probability sequence includes the prediction probability value corresponding to different sign language words. Obtain the real sign language words corresponding to each frame of the training sample, and determine the real sign language word probability sequence based on the real sign language words and the probability distribution sequence corresponding to each sign language word. Based on the real sign language lexical probability sequence and the predicted sign language lexical probability sequence corresponding to each frame of image, the language perception label smoothing loss function is constructed.

7. The sign language recognition method according to claim 6, characterized in that, The language-aware label smoothing loss function is constructed based on the real sign language lexical probability sequence and the predicted sign language lexical probability sequence corresponding to each frame image, including: according to Construct the language-aware label smoothing loss function; in, This represents the language-aware label smoothing loss function. Indicates the total number of sign language lexical units. Indicates the first The probability sequence of real sign language lexical units corresponding to the frame image. This represents the probability sequence of predicted sign language morphemes corresponding to the current frame image. The predicted probability value corresponding to each sign language word.

8. The sign language recognition method according to claim 4, characterized in that, The process of training the sign language recognition model based on the language-aware label smoothing loss function to obtain a trained sign language recognition model includes: For the training samples in the training process of the sign language recognition model, obtain the sign language recognition result output by the sign language recognition model of the training samples, and the visual output tensor determined by the visual branch in the sign language recognition model; Based on the sign language recognition results, a main classification loss function is constructed; Based on the visual output tensor, the visual feature recognition result is determined, and based on the visual feature recognition result, a visual branch-assisted classification loss function is constructed. The main classification loss function, the visual branch auxiliary classification loss function, and the language-aware label smoothing loss function are weighted and fused to obtain the total loss function; The sign language recognition model is trained using the total loss function to obtain a trained sign language recognition model.

9. A sign language recognition device, characterized in that, include: An acquisition module is used to acquire a sequence of image frames to be identified and to determine the skeleton detection result based on the sequence of image frames; The recognition module is used to input the image frame sequence and the skeleton detection result into a pre-trained sign language recognition model to obtain the sign language recognition result corresponding to the image frame sequence; The sign language recognition model includes a cross-modal attention fusion module; The cross-modal attention fusion module is used to calculate a first semantic enhancement result of the visual feature vector on the skeletal feature vector, and a second semantic enhancement result of the skeletal feature vector on the visual feature vector, and to determine a frame-level prediction probability sequence based on the first semantic enhancement result and the second semantic enhancement result; the frame-level prediction probability sequence is used to determine the sign language recognition result; The visual feature vector is obtained by feature extraction based on the image frame sequence; the skeletal feature vector is obtained by feature extraction based on the skeletal detection results.

10. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method described in any one of claims 1-8.

Images