Sign language translation method and device based on vision and word feature pre-training alignment

Abstract

The invention discloses a sign language translation method and device based on vision and word feature pre-training alignment, and belongs to the technical field of sign language translation.The sign language translation method comprises the steps that vision features of sign language videos are extracted through a vision encoder, and word text features are extracted through a text encoder; performing comparative learning based on vision and word text features to obtain a pre-trained vision encoder; performing word text mask prediction pre-training on the text decoder; constructing the pre-trained visual encoder and the pre-trained text decoder into a sign language recognition model so as to recognize a word text sequence from the sign language video; and accessing the pre-trained large language model in the domain into a sign language recognition model to construct a sign language translation model, and performing combined fine tuning so as to translate the word text sequence into a natural language text. The sign language recognition and translation method can realize more efficient, accurate and reliable sign language recognition and translation, is applied to the fields of intelligent sign language translation, barrier-free communication, sign language education and the like, and provides more accurate and natural language interaction experience for hearing-impaired groups.

Description

Technical Field

[0001] The present invention belongs to the technical field of sign language translation, and in particular relates to a sign language translation method and device based on pre-training alignment of visual and word features. Background Art

[0002] As the main means of communication among people with hearing impairments, sign language plays a vital role in information exchange and social interaction. With the rapid development of information technology and artificial intelligence, sign language recognition and translation technology has increasingly become an important technical means to build a barrier-free society. Sign language recognition can be divided into the following two categories according to different tasks. The first category is sign language gloss recognition (ISLR), which usually refers to the recognition of a single sign language gloss or a single character from a sign language video; the second category is continuous sign language recognition (CSLR), which aims to recognize a sequence of sign language glosses with the same word order as the sign language video from a sign language video. Sign language translation is a more complex task, which requires the combination of computer vision perception technology to perceive the deep features corresponding to the sign language video image and natural language processing to understand the text information corresponding to the sign language video. According to different research paradigms, the sign language translation framework can be divided into two types: sign language video to text (sign2-text, S2T) and sign language video to gloss to text (sign2gloss2text, S2G2T). The S2T framework directly translates continuous sign language videos into natural language sentences, while the S2G2T framework first extracts word sequences from sign language videos through a continuous sign language recognition model, and then uses the pre-trained Gloss2Text network to translate the word sequences into natural language text.

[0003] In terms of cutting-edge technology, Gloss-free algorithms and Gloss-based algorithms each demonstrate unique advantages. Gloss-free algorithms usually use a combination of visual encoders and text decoders to optimize model performance through pre-training and joint training. This type of algorithm continues to innovate in model structure, such as using combinations such as ViT+GPT, ViT+T5, or CLIP+Transformer to better capture the deep features of sign language videos and generate accurate natural language text. In addition, unsupervised algorithms and training enhancement strategies are also widely used to improve the generalization ability and translation accuracy of the model. Although the Gloss-free algorithm reduces the reliance on sign language words, it is difficult to align visual features with text features in the absence of clear word supervision.

[0004] Gloss-based algorithms use the supervised information of words to strengthen the connection between sign language videos and natural language. This type of algorithm usually adopts the framework structure of visual encoder and text decoder, and combines the connection temporal classification (CTC) loss and text generation loss to achieve joint training of sign language recognition and translation. During the training process, a variety of enhancement strategies are used to improve model performance, such as using external corpus for pre-training, step-by-step optimization of visual and translation models, and cross-task learning. In addition, technologies such as fusing skeleton key points, introducing multimodal features, optimizing model structure, and introducing new loss functions are also used to further improve the performance of Gloss-based algorithms. Although the Gloss-based algorithm uses word supervision information, the accurate extraction of word sequences in continuous sign language recognition still faces challenges. In addition, due to the high cost of collecting and annotating word data, data scarcity problems arise, which further limits the generalization ability of the translation model.

[0005] In summary, due to the complexity and diversity of sign language videos, existing methods still have the problem of insufficient alignment of visual features and word features in the process of sign language video recognition and translation, which in turn affects the alignment from word to text, making the accuracy of sign language recognition and translation need to be further improved. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a sign language translation method and device based on pre-training alignment of visual and word features, and feature learning is performed through two core pre-training tasks: one is to compare and learn visual features with word text features, aiming to align the semantic representation of sign language video and word text; the other is a mask prediction task based on word text, aiming to enhance the generation ability of word text. In visual feature learning, human posture features and motion characteristics are fully integrated to fully capture the fine-grained motion information and global semantic relationships in sign language videos, aiming to improve the accuracy of sign language video recognition. After pre-training is completed, the language translation capability of a large language model is used based on words to achieve end-to-end training from sign language video to natural language, which effectively solves the bottleneck problems of existing methods in semantic alignment and temporal modeling, and significantly improves the translation accuracy and robustness of the model, thereby achieving more efficient, accurate and reliable sign language recognition and translation services. The present invention has the characteristics of strong versatility, high training efficiency and excellent performance. It provides an innovative solution for the development of sign language translation technology and can be widely used in intelligent sign language translation, barrier-free communication, sign language education and other fields, providing the hearing-impaired group with a more accurate and natural language interaction experience.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows: An embodiment of the present invention provides a sign language translation method based on pre-training alignment of visual and word features, comprising the following steps: A visual encoder is used to extract gesture features and motion features from the sign language video and fuse them into visual features. A text encoder is used to extract word text features from the original word text. Comparative learning is performed based on visual features and word text features to obtain a pre-trained visual encoder. Using the text encoder and the text decoder to pre-train the mask prediction task based on the original word text to obtain a pre-trained text decoder; The pre-trained visual encoder and the pre-trained text decoder are constructed into a sign language recognition model to recognize word text sequences from the sign language video, and the sign language recognition model is trained; The large language model pre-trained in the domain based on the public dataset and the constructed sign language translation dataset is used as the language translation model to convert the word text sequence into natural language text. The language translation model is connected to the trained sign language recognition model to construct a sign language translation model and jointly fine-tuned to achieve the translation of sign language videos into natural language text. The new sign language video is input into the jointly fine-tuned sign language translation model to perform sign language translation and output natural language text.

[0008] Preferably, the step of extracting gesture features and motion features from the sign language video using a visual encoder and fusing them into visual features comprises: A visual encoder is constructed including the Sapiens large model, the VideoMAE large model and the feature fusion network. The Sapiens large model is used to capture the dynamic changes of the signer in the spatial dimension to extract the posture features of the human body from the sign language video. The posture features include the signer's skeletal structure, body movements and facial expressions. The VideoMAE large model is used to capture the dynamic changes of the signer in the time series dimension to extract the motion features of the human body from the sign language video. The motion features include the motion trajectory, gesture changes and rate features between video frames. The feature fusion network is used to fuse the posture features and motion features to obtain the visual features of the sign language video.

[0009] Preferably, feature fusion is performed in the feature fusion network, including: The posture features and motion features are respectively mapped to the same feature dimension space through the posture feature linear layer and the motion feature linear layer to achieve feature dimension unification. The posture features and motion features after dimension unification are concatenated on the feature dimension to obtain a new feature vector. The new feature vector is input into a two-layer convolutional neural network for high-dimensional feature extraction. Each convolutional neural network includes a convolution layer, a normalization layer, an activation layer and a pooling layer. The high-dimensional features extracted by the two-layer convolutional neural network are input into a multi-layer perceptron for further feature fusion and transformation to obtain visual features with rich semantic information.

[0010] Preferably, the comparative learning based on visual features and word text features to obtain a pre-trained visual encoder includes: The CLIP algorithm is used to construct the CLIP loss function by calculating the similarity between visual features and word text features to achieve feature alignment at the video and sentence levels. The SoftDTW algorithm is used to construct the SoftDTW loss function by calculating the cost matrix between visual features and word text features to achieve feature alignment at the continuous video frame and word level. The CLIP loss function and the SoftDTW loss function are used as the total alignment loss function in contrastive learning to pre-train the visual encoder to obtain the pre-trained visual encoder.

[0011] Preferably, the pre-training of the mask prediction task based on the original word text by using the text encoder and the text decoder to obtain the pre-trained text decoder includes: The original word text is randomly masked to obtain masked word text, the masked word text is input into the text encoder for encoding to obtain encoded word text, and then the encoded word text is input into the text decoder for decoding to obtain decoded word text, the text encoder parameters are frozen, the original word text is used as a label and the text decoder is pre-trained through a mask prediction loss function, and the generation ability of the text decoder is trained by reconstructing the mask content to obtain a pre-trained text decoder.

[0012] Preferably, both the text encoder and the text decoder are constructed using the mBART model pre-trained on a large-scale corpus covering multiple languages.

[0013] Preferably, the large language model pre-trained in the domain based on the public dataset and the constructed sign language translation dataset is used as the language translation model, including: Construct a word-based sign language translation dataset that strictly follows the standard of universal sign language dictionaries and covers multi-domain sign language words. It provides a correspondence between words and natural languages. In the sign language translation dataset, each word text sequence corresponds to a natural language sentence. Based on the acquired public dataset and the constructed sign language translation dataset, the large language model is pre-trained in the domain through the Lora fine-tuning method to obtain a language translation model.

[0014] Preferably, the large language model adopts the Qwen model.

[0015] Preferably, the sign language recognition model is trained and the sign language translation model is jointly fine-tuned using the data-enhanced sign language video dataset, and the data enhancement includes: cropping, rotating, translating, and adjusting the color, brightness, contrast, and sharpening effects of the sign language video, and then normalizing the pixel values.

[0016] To achieve the above-mentioned purpose of the invention, the embodiment of the present invention further provides a sign language translation device based on pre-training alignment of visual and word features, which is implemented by the above-mentioned sign language translation method based on pre-training alignment of visual and word features, and includes: an alignment pre-training module, a mask pre-training module, a sign language recognition training module, a language translation fine-tuning module, and a sign language translation module; The alignment pre-training module is used to use a visual encoder to extract gesture features and motion features from the sign language video respectively and fuse them into visual features, use a text encoder to extract word text features from the original word text, and perform comparative learning based on the visual features and word text features to obtain a pre-trained visual encoder; The mask pre-training module is used to perform pre-training of a mask prediction task based on the original word text using a text encoder and a text decoder to obtain a pre-trained text decoder; The sign language recognition training module is used to construct a sign language recognition model by using a pre-trained visual encoder and a pre-trained text decoder to recognize a word text sequence from a sign language video, and to train the sign language recognition model; The language translation fine-tuning module is used to use a large language model pre-trained in the domain based on a public data set and a constructed sign language translation data set as a language translation model to convert a word text sequence into a natural language text, connect the language translation model to the trained sign language recognition model to construct a sign language translation model and perform joint fine-tuning to achieve translation of sign language videos into natural language texts; The sign language translation module is used to input a new sign language video into the jointly fine-tuned sign language translation model to perform sign language translation and output natural language text.

[0017] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention combines human posture features and motion features to construct a feature fusion network, enhances the model's ability to capture the sign language speaker's spatial position information and temporal relationship information, comprehensively captures the fine-grained motion information and global semantic relationship in the sign language video, and helps to improve the accuracy of sign language recognition. In addition, the present invention proposes a sign language recognition and translation architecture based on word-to-natural language after pre-training alignment of visual and word features. The architecture combines the pre-training alignment method of visual and word features with the method of sign language translation based on words, improves the overall performance and robustness, and can achieve more efficient, accurate and reliable sign language recognition and translation.

[0018] (2) The present invention adopts two-stage pre-training. In the first stage, the visual encoder is pre-trained by comparative learning of visual features and word text features, and the sign language video features are aligned with the word features. The video and word text are aligned through a common vector space, so that the visual encoder can efficiently perform cross-modal understanding to adapt to the sign language recognition task. In the second stage, the text decoder is pre-trained by mask prediction based on the original word text, which further enhances the word text generation capability, so that the constructed sign language recognition model has accurate recognition capability from video to word text.

[0019] (3) The present invention uses CLIP loss and SoftDTW loss to optimize feature alignment in the comparative learning stage based on visual and word text features, accurately matching the dynamic movements of sign language videos with word text semantics, making the video information highly consistent with the language expression, thereby improving the quality of sign language recognition.

[0020] (4) The present invention provides a high-quality correspondence between words and natural languages ​​by constructing a word-based sign language translation dataset. It reduces the dependence on large-scale annotated data through in-domain pre-training and enhances the language translation and generalization capabilities of large language models. By connecting a sign language recognition model to a sign language translation model and performing joint fine-tuning, it achieves accurate translation from sign language video to word text sequence and finally to natural language text. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a flow chart of a sign language translation method based on pre-training alignment of visual and word features provided by an embodiment of the present invention; Figure 2 is a schematic diagram of a feature fusion network provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a framework based on pre-training alignment of visual and word features provided by an embodiment of the present invention; Figure 4 It is a schematic diagram of a data path from vocabulary to training in the natural language domain based on a large language model provided by an embodiment of the present invention; Figure 5 It is a schematic diagram of the overall framework of the sign language translation stage provided by an embodiment of the present invention; Figure 6It is a structural schematic diagram of a sign language translation device based on pre-training alignment of visual and word features provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0024] The inventive concept of the present invention is: in view of the problem that insufficient alignment of visual features and word features still exists in the process of sign language video recognition and translation in the prior art, resulting in insufficient translation accuracy, the embodiment of the present invention provides a sign language translation method and device based on pre-training alignment of visual and word features. For a sign language video data set, a combination of human posture features and motion characteristics is adopted to realize sign language recognition after pre-training alignment of visual features and word text features, and then language translation is realized based on word text sequences to natural language texts through a large language model. The sign language recognition and language translation processes are constructed as a sign language translation framework as a whole, and combined with data enhancement, feature fusion, mask text prediction and other optimization technologies, the semantic representation ability and translation accuracy of the sign language video are effectively improved, the deficiencies of the prior art in feature modeling and semantic alignment are overcome, and accurate and fluent translation from sign language video to natural language is realized, providing an efficient and robust solution for sign language translation.

[0025] The goal of Sign Language Translation (SLT) is to accurately predict the corresponding natural language sentence given a sequence of sign language video frames. Input: contains Frames of sign language video = ,in For the Frame video, and The length and width of the video frame. Output: Contains natural language sentence with word segments = ,in For the A participle.

[0026] Figure 1 FIG. 1 is a flow chart of a sign language translation method based on pre-training alignment of visual and word features provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a sign language translation method based on pre-training alignment of visual and word features, comprising the following steps: S1, use the visual encoder to extract gesture features and motion features from the sign language video respectively and fuse them into visual features, use the text encoder to extract word text features from the original word text, and perform comparative learning based on visual features and word text features to obtain the pre-trained visual encoder.

[0027] S1.1, sign language video data enhancement preprocessing. The sign language video of each frame is saved in the format of an image and the RGB value is normalized to the range of [0, 1]. During the training phase, the input sign language video is horizontally flipped in the spatial dimension with a probability of 50%, including cropping, rotation, translation, and adjustment of color, brightness, contrast, and sharpening effects. Then, the pixel values ​​of the input image data are normalized, the mean and standard deviation of each channel (RGB) of the image are defined, and each pixel value is standardized according to the following formula: , in, is the normalized pixel value, is the original pixel value, and They are the mean and standard deviation of the pixel values, respectively. The pixel values ​​are centered by subtracting the mean to make their mean 0, eliminating the influence of brightness differences. The pixel value range is normalized by dividing by the standard deviation to avoid unstable model optimization caused by eigenvalues ​​at different scales. Standardization can accelerate the convergence of the model, avoid the problem of gradient disappearance or explosion, and improve training stability.

[0028] Generating diverse training samples through these data enhancement methods can improve the robustness and generalization ability of the dataset, provide rich data support for subsequent model training, and avoid overfitting problems.

[0029] S1.2, Human body posture feature encoding. Use the pre-trained Sapiens large model as the human body posture feature encoder From the sign language video Extract human posture features from the Sapiens model, which include the signer's skeleton structure, body movements, and facial expressions. Freeze the parameters of the Sapiens model and extract posture features through the Sapiens model's ability to capture spatial features. , is the number of video frames, for The characteristic dimension of is defined as: , in, It is a human posture feature encoder.

[0030] In the embodiment, the Sapiens large model is based on the Masked Autoencoder (MAE) architecture, and draws on the image encoding method of the Vision Transformer (ViT). By pre-training on the large-scale dataset Humans-300M, it can extract fine-grained representations of gestures, arms, upper body and other parts from the image, including the posture, contour, structure and relationship information of the human body. These deep representations provide visual information of the signer's posture, and by capturing the dynamic changes of the signer in space, the model's ability to understand the sign language expression is improved, and the visual encoding process is optimized. The encoder of the Sapiens large model processes the image by dividing the image into non-overlapping image blocks of fixed size. Unlike the traditional convolutional neural network (CNN), the Sapiens large model adopts a self-attention mechanism, which can capture the long-distance dependencies and global information in the image, randomly select a part of the image blocks for masking, and the remaining image blocks remain visible. Through this masking strategy, the model can learn to recover the complete image from partially visible information, thereby effectively extracting the potential representation of the image. In the embodiment, the frozen Sapiens large model is used to extract the features of the signer. First, the video of the signer is As input, each frame image is interpolated to a size of 1024×1024 pixels. Then, the image is divided into fixed-size 16×16 image blocks. , where the number of image blocks seq is 4096, In this process, no image block is masked in order to fully extract the potential human representation of the image. The sequence is input into the ViT architecture, and the feature dimension after the linear projection layer is And add it with the position-encoded features, the feature dimension of the image block is mapped to , the feature size obtained after passing through the multi-layer Transformer encoder in Vit is , where the model size is 0.3b, 0.6b, 1b and 2b according to the different parameters of the Spaiens model The pixels are 1024, 1280, 1536 and 1920 respectively, and the number of Transformer layers is 24, 32, 40 and 48 respectively. The features are obtained by mean pooling. , and finally T The features of the frames are spliced ​​to obtain the human posture features of the sign language video .

[0031] S1.3, Human motion feature encoding. Use the pre-trained VideoMAE large model as the human motion feature encoder From the sign language video The motion features include motion trajectory, gesture change and rate features between video frames to supplement the insufficiency of posture features. The parameters of the VideoMAE large model are frozen to process the sign language segments segmented from the video. The sliding window method is used to capture the implicit word-level representation. The sign language video is first segmented into overlapping segments of fixed length, and then each segment is input into , extracting implicit word-level motion features ,in N= T / t , t is the number of frames between the start frames of adjacent segments, for The characteristic dimension of Yes means N The feature sequence at the fragment level is defined as: , in, It is a human motion feature encoder.

[0032] In the embodiment, the VideoMAE large model is pre-trained on the Kinetics-400 dataset, and uses the Masked Autoencoder technology to model the time series of sign language videos, extract the motion trajectory, gesture changes and rate characteristics between video frames, and enhance the time series modeling ability of sign language actions, so that the translation system can more accurately capture and understand the dynamic information in the sign language video. In the embodiment, by freezing its parameters, it is used to extract the motion features of the sign language video. The input video is represented as , where each frame is cropped to a fixed size First, we use a sliding window of size 16 frames to divide the input video into N video clips, the step length of the sliding window is t , N= T / t , a single video is divided into N Clips ,in , VideoMAE uses a spatiotemporal block embedding of size 2×16×16×3 to sample the video frames, and the obtained feature representation is , where seq is the sequence length, which is 1568 ( ), and the dimension of each spatiotemporal block is mapped to The final encoding feature is , and then pool the feature into , and finally N The features of the fragments are spliced ​​together to obtain .

[0033] S1.4, Feature Fusion. Feature Fusion Network For fusion of posture features and motion characteristics .like Figure 2 As shown in the figure, the posture feature and motion feature are mapped to the same feature dimension space through the posture feature linear layer and the motion feature linear layer respectively, so as to unify the feature dimension and obtain the posture feature. and motion characteristics ,in The unified feature dimension is set to 1024 in the embodiment. and motion characteristics Concatenate on the feature dimension to get a new feature vector , , expressed as: , in, Represents feature concatenation.

[0034] The concatenated feature vector Input into a two-layer convolutional neural network for high-dimensional feature extraction. Each layer of convolutional neural network (CNN) includes a convolution layer (K5), a normalization layer (BN), an activation layer (ReLU) and a pooling layer (P2). The structure of the two-layer CNN is expressed as {K5+BN+ReLU→P2→K5+BN+ReLU→P2}, where the convolution kernel size of the convolution layer is 5 and the step length is 1, and the pooling layer is a maximum pooling layer with a pooling kernel size of 1 and a step length of 2. By extracting short-term spatiotemporal features, the maximum pooling layer is used to reduce the length of the feature sequence. The high-dimensional features extracted by the two-layer convolutional neural network are input into a multi-layer perceptron (MLP) for further feature fusion and transformation to obtain visual features with rich semantic information. ,in is the length of the sequence after feature fusion. The above feature fusion strategy can effectively retain the spatial structure information and temporal dynamic characteristics of gesture actions, thereby optimizing the overall representation ability of sign language videos.

[0035] S1.5, text feature encoding. In order to effectively process natural language text data, the text encoder in the pre-trained mBART model is used The mBART model is a multilingual neural machine translation (NMT) model based on an autoregressive model. It has been pre-trained on the large-scale CC25 corpus containing 25 languages. Its encoder consists of a 12-layer Transformer stack, which can effectively capture the semantic and contextual information of the input sentence. Text Encoder Original word text Extract semantic information and effectively encode language features to obtain word text features , is the number of words, for The characteristic dimension of is defined as: , in, A text encoder.

[0036] S1.6, alignment of visual features and word text features. Figure 3 As shown in the figure, the Sapiens large model, the VideoMAE large model and the feature fusion network are constructed as a visual encoder. The CLIP algorithm and the SoftDTW algorithm are used simultaneously to align visual features and word text features to enhance the model's ability to model the semantics and temporal dependencies of sign language videos, improve the matching accuracy of sign language video actions and text words, and thus optimize the accuracy of sign language translation. Among them, the CLIP algorithm constructs the CLIP loss function by calculating the similarity between visual features and word text features to achieve feature alignment at the video and sentence levels, and the SoftDTW algorithm constructs the SoftDTW loss function by calculating the cost matrix between visual features and word text features to achieve feature alignment at the level of continuous video frames and word levels. The CLIP loss function and the SoftDTW loss function are used as the total alignment loss function in contrastive learning to pre-train the visual encoder, thereby obtaining a pre-trained visual encoder.

[0037] The core idea of ​​the CLIP algorithm is to learn a visual-language representation model through pre-training of large-scale image-text pair data, which can map images and texts to the same semantic space, so that images and texts can be efficiently retrieved and understood across modalities. CLIP emphasizes the advantages of learning from natural language over other task-independent pre-training methods, making it particularly suitable for sign language tasks. The characteristic of CLIP is that it uses a contrastive learning method to represent visual features and word text features in the same semantic space, and optimizes model parameters by calculating the cosine similarity between visual features and word text features. The training goal of CLIP is to maximize the cosine similarity of positive sample pairs (i.e., samples with matching visual features and word text features) and minimize the cosine similarity of negative sample pairs (i.e., samples with mismatched visual features and word text features). This method can effectively enhance the similarity between sign language videos and corresponding texts in the semantic space, ensure that the visual features of sign language videos can accurately express the meaning of natural language, and achieve global alignment at the video level and text level. and word text features The input is sent to each head network and mapped to the joint multimodal semantic space through linear projection for similarity calculation. In CLIP, both head networks consist of a linear layer, and this process is expressed as: , in, is a linear layer, The last layer of the visual encoder is <cls>The activation value at The last layer of the text encoder is <eos>The activation value at . Then, and is layer-normalized and pair-wise scaled to generate a pair of video and sentence features { }, and then calculate the CLIP loss through the symmetric cross entropy loss function , its formula is expressed as: , in, and Respectively represent The visual features and The text features of the word, and Respectively represent The visual features and The text features of the word, represents the cosine similarity, Indicates the size of the batch, represents the learnable scaling factor.

[0038] The SoftDTW algorithm is an improved version of the Dynamic Time Warping (DTW) algorithm, which uses a flexible alignment strategy to handle potential timing differences in video and word text sequences. The algorithm is suitable for calculating the distance between two sequences of different lengths: visual features and word text features. According to the continuity and unidirectional characteristics of the alignment path, it can effectively reflect the similarity between two time series. This method performs fine-grained alignment of the key frame sequence of the sign language video with the corresponding text words, so that the temporal dynamic changes of the sign language movements can more accurately match the text sequence, and can solve the problem of differences in gesture execution speed and movement style among different signers, ensuring that the model can capture the temporal dependence of sign language expression, thereby optimizing the correspondence between video frames and text words, and achieving accurate alignment at the image level and word level. By summing the distance of each point on the alignment path, the dynamic time warping distance obtained is used as the DTW distance to measure the similarity of time series, which can be back-propagated during the neural network training process to optimize the network parameters. In the embodiment, the visual feature is input and word text features Two feature sequences, SoftDTW introduces a smooth distance function to measure each pair of elements ( The distance between , the cost matrix is ​​calculated as follows: , , in, For the visual features, No. The text features of the word, is the Euclidean distance, is the Softmin function, is the cost matrix Line Matrix elements of columns, , and are used in formulas to refer to , and .parameter Used to control the smoothness of the alignment. , approaches the Euclidean distance, and when When , it approaches the traditional DTW. The ablation experiment was carried out with the value of .

[0039] Finally, the total alignment loss function in contrastive learning is It is expressed as: ,

[0040] in, is the cost matrix Line Matrix elements of columns. Based on the total alignment loss function The visual encoder is optimized through contrastive learning to obtain a pre-trained visual encoder.

[0041] S2, using the text encoder and text decoder to perform pre-training on the mask prediction task based on the original word text to obtain a pre-trained text decoder.

[0042] In the embodiment, the text encoder and text decoder in the mBART model pre-trained on a large-scale corpus containing multiple languages ​​are used, and the text sequence containing the word is used. The Mbart word segmenter is pruned based on the dataset. The text encoder and text decoder are trained using a masked self-supervised learning strategy, with the goal of recovering the masked words in the input sentence. According to the name of the trained sign language video, the corresponding word text is found and the text is masked. In the embodiment, a text perturbation strategy based on noise injection is used to enhance the robustness of the text decoder by simulating a noisy environment. Specifically, first, some words in the original word text are randomly discarded according to a preset noise rate, and special markers are used to mark the text. <mask>Replace, get the mask word text, these missing positions through the noise rate Dynamic selection is performed, thus introducing uncertainty and randomness. For the input word text sequence , retain the total The first word in words, The calculation formula is: , in, is A uniformly distributed random value in the range replaces the non-reserved words in the word text sequence with <mask>, get the masked word text. Figure 3 As shown in the figure, the masked word text is segmented by the word segmenter and then input into the text encoder (using a 12-layer Transformer structure) for encoding to obtain the encoded word text, and then the encoded word text is input into the text decoder for decoding to obtain the decoded word text, freeze the text encoder parameters, use the original word text as the label and pass the mask prediction loss function (i.e., the cross entropy loss function) is used to pre-train the text decoder, and the generation ability of the text decoder is trained by the above-mentioned reconstructed mask content to obtain the pre-trained text decoder.

[0043] S3, constructing the pre-trained visual encoder and the pre-trained text decoder into a sign language recognition model to recognize word text sequences from the sign language video, and training the sign language recognition model.

[0044] In the sign language recognition stage (SLR), the pre-trained visual encoder and pre-trained text decoder are constructed as a sign language recognition model to convert the input sign language video frame sequence into the corresponding word text sequence.

[0045] Pre-trained visual encoder The task is to Perform feature extraction to obtain visual features The visual encoder has been optimized for the representation of visual information by co-training with the text decoder in the pre-training stage. The language classifier and CTC loss are used for training after the visual encoder. CTC loss considers all possible alignment paths between visual features and word text sequences while minimizing the error. By traversing the input video and the corresponding word text sequence All possible alignment paths between them are used to calculate the conditional probability : , in, For one of the paths, For all possible alignment paths, the probability The CTC loss is calculated by the language classifier (independent of the visual encoder, including linear and softmax layers to map visual features to frame-level word probabilities). It is expressed as: .

[0046] Pre-trained text decoder The task is based on visual features Generate the corresponding word text sequence The text decoder will gradually predict the next word segmentation , until a complete word text sequence is generated. This process is based on modeling the conditional probability of the previous word segmentation, and the formula is expressed as: , , , The first word of the sentence is artificially set as a special marker word. <bos>, until the marker word is generated <eos>, the text decoder will end up generating the decoded vector In the linear layer and the Softmax layer Then generate the predicted word text sequence , is the weight of the linear layer, is the bias of the linear layer. By minimizing the cross entropy loss of the generated word text sequence To optimize the network.

[0047] In the sign language recognition stage, the total optimization loss for training the sign language recognition model It is expressed as: , in, and is the weight parameter, in the embodiment .

[0048] S4, uses the large language model that has been pre-trained in the domain based on the public dataset and the constructed sign language translation dataset as the language translation model to convert the word text sequence into natural language text. The language translation model is connected to the trained sign language recognition model to construct a sign language translation model and perform joint fine-tuning to achieve the translation of sign language video into natural language text.

[0049] In the sign language translation stage (STR), sign language translation is implemented based on pre-trained alignment of video and word text features, and the translation scheme of word text sequences to natural language sentences based on fine-tuning of a large language model is further explored. This not only reduces the dependence on large-scale annotated data, but also improves the effect and scalability of sign language translation tasks.

[0050] In the embodiment, the large language model uses Qwen2.5-14B-Instruction, which is used as a language translation model after in-domain pre-training on the constructed sign language translation dataset and the public dataset through Lora fine-tuning, so that the model can effectively capture the semantic association between the words in the sign language video and the natural language, such as Figure 4 As shown in , the translation from the word text sequence to the natural language sentence is realized. Figure 5 As shown, after the large language model (language translation model) pre-trained in the domain is connected to the trained sign language recognition model, the whole is constructed as a sign language translation model and fine-tuned jointly to achieve word-based sign language translation. The word-based sign language translation dataset constructed in the embodiment strictly follows the standard of the general sign language dictionary, covers multi-field sign language words, provides a correspondence between words and natural languages, has wide coverage and accurate annotations, and each word text sequence in the sign language translation dataset corresponds to a natural language sentence, providing high-quality data support for model training. By performing in-domain pre-training on self-built sign language translation datasets and public datasets and combining fine-tuning of the large language model, language translation from sign language words to natural languages ​​can be achieved, which can reduce the dependence on large-scale annotated data and improve the generalization ability of sign language translation tasks.

[0051] The process of the language translation stage is expressed as: , , in, is the language translation model, For the sequence of words from the text Generate natural language sentences The conditional probability of generating natural language sentences by minimizing the cross entropy loss To optimize the network, it is expressed as follows: .

[0052] In the sign language translation stage, the total optimization loss for jointly fine-tuning the sign language translation model including the language translation model and the trained sign language recognition model It is expressed as: , in, and is the weight parameter, in the embodiment .

[0053] S5, input the new sign language video into the jointly fine-tuned sign language translation model to perform sign language translation and output natural language text.

[0054] The goal of sign language translation is to convert the input sign language video frames The sequence is converted into the corresponding natural language sentence Specifically, the sign language video frames are input into the jointly fine-tuned sign language recognition model to obtain the predicted word text sequence, and then the word text sequence is mapped into the vocabulary through the jointly fine-tuned language translation model to obtain the natural language sentence formed by the segmented words in the word text sequence.

[0055] Based on the same inventive concept, Figure 6 As shown, an embodiment of the present invention also provides a sign language translation device 600 based on visual and word feature pre-training alignment, including: an alignment pre-training module 610, a mask pre-training module 620, a sign language recognition training module 630, a language translation fine-tuning module 640, and a sign language translation module 650.

[0056] The alignment pre-training module 610 is used to use a visual encoder to extract posture features and motion features from the sign language video respectively and fuse them into visual features, use a text encoder to extract word text features from the original word text, and perform comparative learning based on visual features and word text features to obtain a pre-trained visual encoder.

[0057] The mask pre-training module 620 is used to perform pre-training of the mask prediction task based on the original word text using the text encoder and the text decoder to obtain a pre-trained text decoder.

[0058] The sign language recognition training module 630 is used to construct a pre-trained visual encoder and a pre-trained text decoder into a sign language recognition model to recognize word text sequences from sign language videos and train the sign language recognition model.

[0059] The language translation fine-tuning module 640 is used to use the large language model that has been pre-trained in the domain based on the public dataset and the constructed sign language translation dataset as a language translation model to convert the word text sequence into natural language text, connect the language translation model to the trained sign language recognition model to construct a sign language translation model and perform joint fine-tuning to achieve the translation of sign language video into natural language text.

[0060] The sign language translation module 650 is used to input the new sign language video into the jointly fine-tuned sign language translation model to perform sign language translation and output natural language text.

[0061] It should be noted that the sign language translation device based on visual and word feature pre-training alignment provided in the above embodiment belongs to the same inventive concept as the sign language translation method based on visual and word feature pre-training alignment. The specific implementation process is detailed in the embodiment of the sign language translation method based on visual and word feature pre-training alignment, which will not be repeated here.

[0062] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.< / eos> < / bos> < / mask> < / mask> < / eos> < / cls>

Claims

1. A sign language translation method based on pre-training alignment of visual and word features, characterized in that: The following steps are involved: A visual encoder is used to extract gesture features and motion features from the sign language video and fuse them into visual features. A text encoder is used to extract word text features from the original word text. Comparative learning is performed based on visual features and word text features to obtain a pre-trained visual encoder. Using the text encoder and the text decoder to pre-train the mask prediction task based on the original word text to obtain a pre-trained text decoder; The pre-trained visual encoder and the pre-trained text decoder are constructed into a sign language recognition model to recognize word text sequences from the sign language video, and the sign language recognition model is trained; The large language model pre-trained in the domain based on the public dataset and the constructed sign language translation dataset is used as the language translation model to convert the word text sequence into natural language text. The language translation model is connected to the trained sign language recognition model to construct a sign language translation model and jointly fine-tuned to achieve the translation of sign language videos into natural language text. The new sign language video is input into the jointly fine-tuned sign language translation model to perform sign language translation and output natural language text.

2. The sign language translation method based on visual and word feature pre-training alignment according to claim 1 is characterized in that: The method of using a visual encoder to extract gesture features and motion features from a sign language video and fuse them into visual features includes: A visual encoder is constructed including the Sapiens large model, the VideoMAE large model and the feature fusion network. The Sapiens large model is used to capture the dynamic changes of the signer in the spatial dimension to extract the posture features of the human body from the sign language video. The posture features include the signer's skeletal structure, body movements and facial expressions. The VideoMAE large model is used to capture the dynamic changes of the signer in the time series dimension to extract the motion features of the human body from the sign language video. The motion features include the motion trajectory, gesture changes and rate features between video frames. The feature fusion network is used to fuse the posture features and motion features to obtain the visual features of the sign language video.

3. The sign language translation method based on visual and word feature pre-training alignment according to claim 2 is characterized in that: Feature fusion is performed in the feature fusion network, including: The posture features and motion features are respectively mapped to the same feature dimension space through the posture feature linear layer and the motion feature linear layer to achieve feature dimension unification. The posture features and motion features after dimension unification are concatenated on the feature dimension to obtain a new feature vector. The new feature vector is input into a two-layer convolutional neural network for high-dimensional feature extraction. Each convolutional neural network includes a convolution layer, a normalization layer, an activation layer and a pooling layer. The high-dimensional features extracted by the two-layer convolutional neural network are input into a multi-layer perceptron for further feature fusion and transformation to obtain visual features with rich semantic information.

4. The sign language translation method based on visual and word feature pre-training alignment according to claim 1, characterized in that: The method of performing comparative learning based on visual features and word text features to obtain a pre-trained visual encoder includes: The CLIP algorithm is used to construct the CLIP loss function by calculating the similarity between visual features and word text features to achieve feature alignment at the video and sentence levels. The SoftDTW algorithm is used to construct the SoftDTW loss function by calculating the cost matrix between visual features and word text features to achieve feature alignment at the continuous video frame and word level. The CLIP loss function and the SoftDTW loss function are used as the total alignment loss function in contrastive learning to pre-train the visual encoder to obtain the pre-trained visual encoder.

5. The sign language translation method based on visual and word feature pre-training alignment according to claim 1, characterized in that: The method of using a text encoder and a text decoder to perform pre-training on a mask prediction task based on the original word text to obtain a pre-trained text decoder includes: The original word text is randomly masked to obtain masked word text, the masked word text is input into the text encoder for encoding to obtain encoded word text, and then the encoded word text is input into the text decoder for decoding to obtain decoded word text, the text encoder parameters are frozen, the original word text is used as a label and the text decoder is pre-trained through a mask prediction loss function, and the generation ability of the text decoder is trained by reconstructing the mask content to obtain a pre-trained text decoder.

6. The sign language translation method based on visual and word feature pre-training alignment according to claim 1 or 5, characterized in that: Both the text encoder and text decoder are built using the mBART model that is pre-trained on a large-scale corpus covering multiple languages.

7. The sign language translation method based on visual and word feature pre-training alignment according to claim 1, characterized in that: The large language model pre-trained in the domain based on the public dataset and the constructed sign language translation dataset is used as the language translation model, including: Construct a word-based sign language translation dataset that strictly follows the standard of universal sign language dictionaries and covers multi-domain sign language words. It provides a correspondence between words and natural languages. In the sign language translation dataset, each word text sequence corresponds to a natural language sentence. Based on the acquired public dataset and the constructed sign language translation dataset, the large language model is pre-trained in the domain through the Lora fine-tuning method to obtain a language translation model.

8. The sign language translation method based on visual and word feature pre-training alignment according to claim 1 or 7, characterized in that: The large language model uses the Qwen model.

9. The sign language translation method based on visual and word feature pre-training alignment according to claim 1, characterized in that: The sign language recognition model is trained and the sign language translation model is jointly fine-tuned using the data-enhanced sign language video dataset. Data enhancement includes: cropping, rotating, translating the sign language video, adjusting the color, brightness, contrast, and sharpening effects, and then normalizing the pixel values.

10. A sign language translation device based on pre-training alignment of visual and word features, implemented by the sign language translation method based on pre-training alignment of visual and word features according to any one of claims 1 to 9, characterized in that: include: Alignment pre-training module, mask pre-training module, sign language recognition training module, language translation fine-tuning module, and sign language translation module; The alignment pre-training module is used to use a visual encoder to extract gesture features and motion features from the sign language video respectively and fuse them into visual features, use a text encoder to extract word text features from the original word text, and perform comparative learning based on the visual features and word text features to obtain a pre-trained visual encoder; The mask pre-training module is used to perform pre-training of a mask prediction task based on the original word text using a text encoder and a text decoder to obtain a pre-trained text decoder; The sign language recognition training module is used to construct a sign language recognition model by using a pre-trained visual encoder and a pre-trained text decoder to recognize a word text sequence from a sign language video, and to train the sign language recognition model; The language translation fine-tuning module is used to use a large language model pre-trained in the domain based on a public data set and a constructed sign language translation data set as a language translation model to convert a word text sequence into a natural language text, connect the language translation model to the trained sign language recognition model to construct a sign language translation model and perform joint fine-tuning to achieve translation of sign language videos into natural language texts; The sign language translation module is used to input a new sign language video into the jointly fine-tuned sign language translation model to perform sign language translation and output natural language text.

Images