Quick sign language translation method based on dialogue key frame selection

Abstract

The invention relates to a rapid sign language translation method based on dialogue key frame selection, which comprises the following steps of: mapping a previous text in a dialogue to a word feature space by using a word embedding model to obtain statement-level text feature representation; inputting the sign language video frame image into a lightweight CNN, and extracting lightweight visual space feature representation; calculating a correlation score between the text feature representation and the lightweight visual space feature representation, and selecting a key frame by adopting a Top-K method; inputting the key frame sequence into a fine CNN (Convolutional Neural Network), and further extracting fine visual space feature representation; inputting the obtained fine visual spatial feature representation into a Transform model, extracting a time sequence feature of the key frame, and decoding the time sequence feature of the key frame to obtain a sign language translation result; and training model parameters.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and computer vision, and involves feature fusion technology, specifically a fast sign language translation method based on dialogue keyframe selection. Background Technology

[0002] Hearing-impaired individuals face significant challenges in accessing information, communicating with others, and perceiving their environment, posing substantial obstacles to their lives, studies, and work. While sign language recognition has garnered considerable attention from researchers, current technology falls far short of meeting the practical application needs of hearing-impaired individuals. This is primarily because sign language recognition models trained on laboratory data struggle to adapt to complex real-world scenarios.

[0003] Sign Language Translation (SLT) aims to convert continuous sign language videos into spoken or written text. Early SLT tasks [1,2] typically employed a two-stage pipeline structure: first, continuous sign language recognition (SLR) was performed to generate a gloss sequence, and then a statistical or neural machine translation model was used to translate the gloss into target language text. Although gloss-based methods have a clear advantage in intermediate semantic representation, they heavily rely on manually annotated gloss data, which is not only costly to annotate but also prone to error accumulation between the recognition and translation stages.

[0004] To overcome the above limitations, research in recent years has gradually shifted to end-to-end sign language translation methods without gloss. Among them, [3] the encoder-decoder architecture based on Transformer has become the mainstream solution due to its advantages in modeling long temporal dependencies. Camgoz et al. [4] proposed an end-to-end Transformer sign language translation framework, which directly maps visual features to text sequences. Experiments show that competitive performance can still be achieved even with reduced or completely removed gloss supervision. On this basis, researchers further introduced a hierarchical temporal modeling strategy to better characterize the multi-scale motion patterns in sign language videos. The temporal semantic pyramid network proposed by Li et al. [5] alleviates the semantic ambiguity problem of frame-level visual representation by modeling cross-scale and intra-scale temporal dependencies.

[0005] In addition to RGB video-based modeling, keypoint (skeleton)-based sign language translation methods have also received widespread attention in recent years. [6] Pose estimation models are typically used to extract keypoints of the human body, hands, and face as compact visual representations, thereby reducing background interference and improving the ability to model fine-grained movements. Since keypoint features significantly reduce input redundancy, [7] skeleton-driven SLT models show stronger robustness and computational efficiency without gloss settings. In addition, some recent studies have begun to introduce generative modeling ideas, such as [8] a sign language translation framework based on diffusion models, to enhance the expressive power of the latent semantic space and improve translation diversity.

[0006] Sign language videos typically have long time sequences and high redundancy, and directly modeling the complete sequence would result in high computational and storage overhead. Therefore, keyframe selection has become an important means to reduce redundancy and improve efficiency. [9] As a traditional keyframe extraction method, deep features are extracted based on convolutional autoencoders, and the most representative frames are selected by clustering or distance metrics, thereby reducing redundant information while ensuring semantic coverage. Such methods have been applied to the preprocessing stage of sign language videos to accelerate the recognition and translation process.

[0007] In sign language recognition and translation tasks, [6] a random sampling or motion intensity-based frame selection strategy is used to retain key gestures while maintaining consistent input length. In recent years,

[10] a dynamic frame selection module has been directly integrated into the end-to-end model, enabling the model to adaptively select key information frames based on temporal features or attention weights. This type of method improves the expressive efficiency of the model by focusing on frames with dense semantic information.

[0008] In addition to frame-level optimization, lightweight model design is also an important direction for improving the overall system efficiency. To address the computational bottleneck of the Transformer architecture,

[11] proposed a variety of sparse attention and parameter compression strategies to reduce memory usage and computational complexity. In the sign language translation scenario,

[12] a lightweight model for edge devices, by combining dynamic keyframe sampling with an efficient attention structure, maintains good translation performance while significantly reducing inference latency. Such research provides important support for the practical deployment of sign language translation systems on mobile and embedded devices.

[0009] In sign language translation systems, modules such as keyframe selection, hard attention mechanism, and discrete temporal segmentation usually involve non-differentiable operations, which block the standard backpropagation process and limit end-to-end training. To address this issue, various gradient approximation and relaxation optimization methods have been proposed. Among them,

[13] makes the category selection operation differentiable under temperature parameter control by approximating the discrete sampling process as a continuous one, thereby supporting gradient backpropagation.

[0010] Another type of method uses random perturbation and expected gradient estimation to deal with nondifferentiable problems.

[14] By introducing random noise into discrete variables and optimizing the expectation of the objective function, a low-variance gradient estimate can be obtained, thereby supporting gradient-based training. In addition,

[15] parametric techniques are applied to the field of variational inference. The core idea is to represent random variables as deterministic functions of parameters and fixed noise, thereby realizing gradient propagation to random nodes.

[0011] When gradient approximation is difficult to achieve, gradient-free optimization methods also provide a feasible alternative.

[16] Using Natural Evolution Strategies (NES), the gradient information is estimated through the forward function, which can still effectively optimize the model parameters in scenarios where direct backpropagation is not possible.

[0012] Related literature: [1] Vogler C, Metaxas D. Toward scalability in ASL recognition:Breaking down signs into phonemes[C] / / International Gesture Workshop. Berlin,Heidelberg: Springer Berlin Heidelberg, 1999: 211-224. [2] Koller O, Forster J, Ney H. Continuous sign language recognition:Towards large vocabulary statistical recognition systems handling multiplesigners[J]. Computer Vision and Image Understanding, 2015, 141: 108-125. [3] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30. [4] Camgoz N C, Koller O, Hadfield S, et al. Sign languagetransformers: Joint end-to-end sign language recognition and translation[C] / / Proceedings of the IEEE / CVF conference on computer vision and patternrecognition. 2020: 10023-10033. [5] Li D, Xu C, Yu X, et al. Tspnet: Hierarchical feature learningvia temporal semantic pyramid for sign language translation[J]. Advances inNeural Information Processing Systems, 2020, 33: 12034-12045. [6] Kim Y, Kwak M, Lee D, et al. Keypoint based Sign LanguageTranslation without Glosses. arXiv 2022[J]. arXiv preprint arXiv:2204.10511. [7] Zhang J, Wang Q, Wang Q. GFTLS-SLT: Gloss-free transformer basedlexical and semantic awareness framework for multimodal sign languagetranslation[J]. IEEE Transactions on Multimedia, 2025. [8] Moon J H, Park J, Kim J, et al. DiffSLT: Enhancing Diversity inSign Language Translation via Diffusion Model[J]. Pattern RecognitionLetters, 2025. [9] ZHOU Zhou, HAN Fang, WANG Zhijie. Video Key Frame Extraction andOptimization Algorithm for Sign Language Recognition[J]. Journal of EastChina University of Science and Technology, 2021, 47(1): 81-88.

[10] Yeung S, Russakovsky O, Jin N, et al. Every moment counts: Densedetailed labeling of actions in complex videos[J]. International Journal ofComputer Vision, 2018, 126(2): 375-389.

[11] Tay Y, Dehghani M, Abnar S, et al. Long range arena: A benchmarkfor efficient transformers[J]. arXiv preprint arXiv:2011.04006, 2020.

[12] Gan S, Yin Y, Jiang Z, et al. Towards real-time sign languagerecognition and translation on edge devices[C] / / Proceedings of the 31st ACMInternational Conference on Multimedia. 2023: 4502-4512.

[13] Jang E, Gu S, Poole B. Categorical reparameterization withgumbel-softmax[J]. arXiv preprint arXiv:1611.01144, 2016.

[14] Maddison CJ, Mnih A, Teh Y W. The concrete distribution: Acontinuous relaxation of discrete random variables[J]. arXiv preprint arXiv:1611.00712, 2016.

[15] Kingma DP, Welling M. Auto-encoding variational bayes[J]. arXivpreprint arXiv:1312.6114, 2013.

[16] Salimans T, Ho J, Chen X, et al. Evolution strategies as ascalable alternative to reinforcement learning[J]. arXiv preprint arXiv:1703.03864, 2017. Summary of the Invention

[0013] The purpose of this invention is to provide a sign language translation method that can accelerate the model's response speed by utilizing the preceding dialogue content. The technical solution is as follows: A fast sign language translation method based on dialogue keyframe selection includes the following steps: Using a word embedding model, the preceding text in the dialogue is mapped to the word feature space to obtain a sentence-level text feature representation; Input sign language video frame images into a lightweight CNN to extract lightweight visual spatial feature representations; Calculate the correlation score between text feature representation and lightweight visual spatial feature representation, and select keyframes using the Top-K method; The keyframe sequence is input into a fine-grained CNN to further extract fine-grained visual spatial feature representations; The obtained fine visual spatial feature representation is input into the Transformer model to extract the temporal features of key frames, and the sign language translation result is obtained by decoding the temporal features of key frames. Training model parameters.

[0014] Furthermore, during model parameter training, two gradient backpropagation paths are used: The first path involves backpropagating frame feature gradients only for keyframes: During gradient backpropagation, for a video frame that has been acquired, if it is a keyframe, the frame feature gradient of the sign language translation main task loss for the fine visual spatial features of this keyframe is calculated to quantify the contribution of the keyframe features to the translation result; if it is not a keyframe, the video frame is not selected and the gradient is set to 0; the assigned gradient is backpropagated along the original frame features to the lightweight CNN. The second approach ensures that the parameters of the similarity calculation module are effectively updated by assigning correlation score gradients: During backpropagation, for a video frame that has been acquired, the similarity score gradient of the sign language translation main task loss on the fine visual spatial features of the keyframe is extracted. The similarity score gradient is a tensor containing channel and spatial dimensions, which is aggregated into a scalar gradient in the channel and spatial dimensions. The scalar gradient is assigned to the correlation score of the keyframe selected by Top-K. For video frames that are not selected as keyframes, their correlation score gradient is set to 0, thus optimizing the correlation score parameters for keyframe selection.

[0015] Furthermore, the mBART model structure serves as the basic framework for the word embedding model.

[0016] Furthermore, the keyframe acquisition method is as follows: (1) Sort the correlation scores numerically, record the corresponding original index numbers, and obtain the sorted video frame sequence; (2) Determine the length of the keyframe sequence based on the set keyframe selection ratio and the total length of the video frames. K ; (3) Select the first frame from the sorted video frame sequence. K Using an index number, extract the corresponding video frame from the original video sequence to construct a keyframe sequence.

[0017] Furthermore, the proportion of keyframes selected is determined based on translation accuracy and translation speed.

[0018] Furthermore, the Pearson correlation coefficient between video frame features and dialogue features is calculated and used as the basis for determining the keyframe selection.

[0019] Furthermore, the MobileNet model structure is used as the basic framework for lightweight CNNs.

[0020] Furthermore, the ResNet18 model structure serves as the basic framework for a refined CNN.

[0021] Furthermore, a distillation learning mechanism is introduced, using a refined CNN as a teacher network to guide the learning process of the lightweight CNN, thereby improving the feature discrimination ability of the lightweight CNN and enhancing the quality of keyframe extraction. Attached Figure Description

[0022] Figure 1 : This is a flowchart of the fast sign language translation method based on dialogue keyframe selection according to the present invention. Figure 2 : This is a model diagram of a fast sign language translation method based on dialogue keyframe selection.

[0023] Figure 3 The impact of selecting different keyframe ratios on the quality of sign language translation. ρ B4 represents the proportion of keyframes selected, and B4 is the evaluation index for sign language translation effect; the larger the value, the better the effect.

[0024] Figure 4 This section compares the effectiveness of the method proposed in this invention with other sign language translation methods. B1, B2, B3, B4, and ROUGE are evaluation indicators for sign language translation effectiveness, with higher values ​​indicating better results. Detailed Implementation

[0025] The present invention will now be described in conjunction with the accompanying drawings and embodiments. This invention provides a fast sign language translation method based on dialogue keyframe selection. The flowchart of the technical solution of this invention is shown below. Figure 1 As shown.

[0026] Step s1: Use a word embedding model to map the preceding text in the dialogue to the word feature space to obtain sentence-level dialogue features; Step s2: Input the sign language video frame images into a lightweight CNN to extract lightweight visual spatial features; Step s3: Calculate the correlation score between lightweight visual spatial features and dialogue features, and select keyframes using the Top-K method; Step s4: Input the selected keyframes into a fine-grained CNN to generate fine-grained visual spatial features; Step s5: Input the fine visual spatial features obtained in step s4 into the Transformer model, extract the temporal features of the keyframes, and obtain the sign language translation results by decoding the temporal features of the keyframes; Step s6: Train the model parameters.

[0027] (1) This invention proposes a dual-path gradient backpropagation strategy to solve the gradient truncation problem caused by Top-K operation during model training; (2) A refined CNN model is used to guide the parameter learning process of the lightweight CNN in order to improve the feature extraction capability and feature expression quality of the lightweight CNN.

[0028] The specific steps are as follows: (I) Text Feature Representation Extraction Using the mBART model to analyze the dialogue text sequence provided by the listener The text is encoded to extract text features from the dialogue text sequence, resulting in a text feature representation. h D This invention uses, but is not limited to, mBART as a dialogue feature extractor.

[0029] (ii) Lightweight visual spatial feature representation extraction Using the MobileNet model to analyze sign language video frames of deaf people Encoding is performed to extract lightweight visuospatial features from the sign language sequence, resulting in a lightweight visuospatial feature representation. h L This invention uses, but is not limited to, MobileNet as a lightweight CNN framework to extract lightweight visual spatial features of sign language.

[0030] (III) Keyframe Selection This invention selects keyframes by effectively utilizing contextual information provided by the listener during a dialogue. Specifically, it calculates the correlation score between video frame features and dialogue features (this invention uses, but is not limited to, the Pearson correlation coefficient to calculate the correlation score). Keyframes are then obtained as follows: (1) Sort the relevance scores numerically and record the corresponding original index numbers; (2) Determine the length of the keyframe sequence based on the 50% selection ratio and the total length of the video frames. K ; (3) Select the first frame from the sorted video frame sequence. K Index number I Extract the corresponding video frames from the original video sequence according to the index number to construct the keyframe sequence; Note 1: Calculation of correlation score between dialogue features and video frame features To effectively utilize problem information for keyframe selection, a linear transformation is first applied to project the text and video representations into the same dimensional space, ensuring their comparability within a unified semantic space. Subsequently, the Pearson correlation coefficient is used to calculate the correlation between the text and video representations.

[0031] in W and b These represent the projection matrix and the bias term, respectively. z Represents the projected features. Let || · || denote the mean of the projected features, and || · || denote the L2 norm of the tensor.

[0032] (iv) Fine-grained video spatial feature extraction The keyframe sequence extracted in step (iii) is denoted as... Using the RestNet 18 model to analyze keyframe sequences The encoding process is performed to extract the fine visual spatial features of the sign language video frame sequence, resulting in a fine visual spatial feature sequence. h R This invention uses, but is not limited to, ResNet 18 as a sophisticated CNN framework to extract sophisticated visual-spatial features of sign language.

[0033] (v) Sign language translation generation A 1D CNN is used to initially extract temporal features from a refined visual spatial feature sequence. The obtained temporal features are then fed into a Transformer Encoder for global modeling, and finally, a Transformer Decoder is used to autoregressively generate the target sequence, outputting the corresponding text translation result.

[0034] (vi) Model parameter training (1) This invention proposes a dual-path gradient backpropagation strategy to solve the gradient truncation problem caused by Top-K operation during model training; (2) A refined CNN model is used to guide the parameter learning process of the lightweight CNN in order to improve the feature extraction capability and feature expression quality of the lightweight CNN.

[0035] Note 2: Dual-path gradient backpropagation strategy During model parameter training in step s6, given that the Top-K based keyframe selection method used in step s3 is a discrete operation, this invention designs two gradient backpropagation paths to address the gradient truncation problem that occurs during backpropagation. Path 1 aims to reduce the unnecessary computational overhead of redundant frames by backpropagating gradients only to keyframes. In the forward phase, the model pre-stores a set of keyframe indices. I (Obtained by sorting and filtering based on the relevance scores of keyframes to the dialogue question), during gradient backpropagation, if the frame index... i belong I Then calculate the loss of the main task of sign language translation. L SLT For the first i Frame feature gradients of fine visual spatial features in keyframes This gradient quantifies the contribution of keyframe features to the translation result. If... i Not belonging to IIf a video frame is not selected, and is considered redundant or noisy, its gradient is directly set to 0. These assigned gradients are then backpropagated along the original frame features to a lightweight CNN (MobileNet), allowing its parameters to be updated accordingly. This enables the learning of more accurate frame features that match the semantics of the problem, improving the expressive power of keyframes. The process is as follows: Path 2 aims to approximate the gradient truncation problem caused by Top-K discrete selection by assigning gradient values ​​to the relevance score to ensure that the parameters of the similarity calculation module are effectively updated, thereby guaranteeing that the parameters of the similarity calculation module can be effectively updated. During backpropagation, the translation loss is first extracted. L SLT The similarity score gradients (including tensors of channel and spatial dimensions) of the fine visual spatial features of keyframes are aggregated into scalar gradients in both channel and spatial dimensions. g i To overcome the gradient truncation in Top-K discrete operations, the gradient is... g i Assign the value to the relevance score of the keyframe selected by Top-K. s i For video frames not selected as keyframes, their relevance score gradient is set to 0. This allows the assigned gradient to propagate smoothly along the relevance score → linear projection → lightweight CNN backpropagation path, optimizing the relevance score parameters for keyframe selection and thus improving the keyframe selection strategy. The process is as follows: in For the first i The aggregate scalar of frame similarity score gradients. These correspond to the coordinates of the channel, height, and width dimensions of the frame features, respectively. For the video number i The relevance score between frames and dialogue.

[0036] Note 3: Keyframe Feature Representation Optimization Compared to lightweight CNNs, sophisticated CNNs have a larger parameter scale and are able to extract more discriminative spatial features, while lightweight CNNs have a smaller parameter count and their spatial feature extraction capabilities are relatively limited. This invention introduces a distillation learning mechanism, using a sophisticated CNN as a teacher network to guide the learning process of a lightweight CNN, thereby improving the feature discrimination ability of the lightweight CNN and enhancing the quality of keyframe extraction.

[0037] To align the temporal dimensions of fine-grained and lightweight visual-spatial features, this invention represents lightweight visual-spatial features separately.h L and fine visual spatial feature representation h R Aggregated features are obtained using time-averaged pooling. and Then, this invention uses mean squared error (MSE) to calculate feature-level distillation loss, reducing the difference between lightweight visual spatial features and fine visual spatial features, thereby guiding the parameter training of lightweight CNNs through fine CNNs: in and represent the sequence lengths of the lightweight visual spatial feature sequence and the fine visual spatial feature sequence, respectively. B Indicates the batch size.

Claims

1. A fast sign language translation method based on dialogue keyframe selection, comprising the following steps: The word embedding model is used to map the preceding text in the dialogue to the word feature space to obtain a sentence-level text feature representation. Input sign language video frame images into a lightweight CNN to extract lightweight visual spatial feature representations; Calculate the correlation score between text feature representation and lightweight visual spatial feature representation, and select keyframes using the Top-K method; The keyframe sequence is input into a fine-grained CNN to further extract fine-grained visual spatial feature representations; The obtained fine visual spatial feature representation is input into the Transformer model to extract the temporal features of key frames, and the sign language translation result is obtained by decoding the temporal features of key frames. Training model parameters.

2. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, During model parameter training, two gradient backpropagation paths are used: The first path is to backpropagate the frame feature gradient only for keyframes: During the gradient backpropagation process, for a video frame that has been acquired, if it is a keyframe, the frame feature gradient of the sign language translation main task loss for the fine visual spatial features of this keyframe is calculated to quantify the contribution of this keyframe feature to the translation result. If it is not a keyframe, the video frame is not selected and the gradient is set to 0; the assigned gradient is backpropagated along the original features of the frame to the lightweight CNN. The second approach ensures that the parameters of the similarity calculation module are effectively updated by assigning correlation score gradients: During backpropagation, for a video frame that has been acquired, the similarity score gradient of the sign language translation main task loss on the fine visual spatial features of the keyframe is extracted. The similarity score gradient is a tensor containing channel and spatial dimensions, which is aggregated into a scalar gradient in the channel and spatial dimensions. The scalar gradient is assigned to the correlation score of the keyframe selected by Top-K. For video frames that are not selected as keyframes, their correlation score gradient is set to 0, thus optimizing the correlation score parameters for keyframe selection.

3. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, The mBART model structure serves as the basic framework for word embedding models.

4. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, The keyframe acquisition method is as follows: (1) Sort the correlation scores numerically, record the corresponding original index numbers, and obtain the sorted video frame sequence; (2) Determine the length of the keyframe sequence based on the set keyframe selection ratio and the total length of the video frames. K ; (3) Select the first frame from the sorted video frame sequence. K Using an index number, extract the corresponding video frame from the original video sequence to construct a keyframe sequence.

5. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, The proportion of keyframes selected is determined based on translation accuracy and translation speed.

6. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, The Pearson correlation coefficient between video frame features and dialogue features is calculated and used as the criterion for key frame selection.

7. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, The MobileNet model structure serves as the basic framework for lightweight CNNs.

8. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, The ResNet18 model structure serves as the basic framework for a refined CNN.

9. The rapid sign language translation method for selecting dialogue keyframes according to claim 1, characterized in that, A distillation learning mechanism is introduced, using a refined CNN as a teacher network to guide the learning process of a lightweight CNN, thereby improving the feature discrimination ability of the lightweight CNN and enhancing the quality of keyframe extraction.

Images