Sign language video translation method, system and medium

Abstract

The present invention provides a translation method, system and computer-readable storage medium for a sign language video. First, a preset deep feedforward encoding module is used to perform deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predicted sign language vocabulary. Then, a preset deep feedforward decoding module is used to perform deep feedforward inference decoding on the sign language video encoding features to obtain predicted words. The predicted words are connected into predicted sentences according to time steps, and then the predicted sentences and the sign language vocabulary are used as translation information files corresponding to the sign language video. In this way, the model's ability to capture complex sign language features, especially the nonlinear relationship expression of sign language movements, can be significantly enhanced, and the problems of inaccurate semantic expression and omission of detailed information in existing models are solved. The generated text can better retain the semantic details in the sign language video, thereby improving the credibility and accuracy of the translation.

Description

Technical Field

[0001] The present invention relates to the field of natural language technology, and more particularly to a sign language self-translation method, system, and medium for translating a sign language video. Background Art

[0002] Sign language is a unique visual language used by the hearing-impaired that facilitates visual communication by using hand and non-hand information. A perfect sign language translation model can convert visual language into natural language without additional steps. Sign language is a brand-new language with unique language logic, grammar, and semantic structure. Semantic segmentation and sign language understanding are two problems encountered when translating sign language directly into natural language text. In the early and mid-stage fields of artificial intelligence, more research focused on sign language recognition tasks, including isolated and sequential sign language recognition, rather than fully translating sign language videos into the target language. Sign language recognition recognizes sign language videos as text and is the written form of sign language.

[0003] The goal of sign language translation is to realize the conversion of sign language video into natural language text, thereby building a bridge for communication between the hearing-impaired and society.

[0004] However, existing sign language translation methods have obvious shortcomings in capturing the complex features of sign language movements, especially the expression of non-linear relationships, resulting in limited accuracy of semantic expression; secondly, most models only support generating natural language text from sign language videos and lack support for auxiliary outputs such as sign language vocabulary, which limits the model's applicability in multiple scenarios; in addition, existing models fail to fully incorporate the unique needs of sign language translation in structural optimization and loss function design, making it difficult to significantly improve translation performance and robustness.

[0005] Therefore, there is an urgent need for a translation method, system and medium for sign language videos that can simultaneously generate sign language vocabulary and sentences, enhance the ability to capture the complex features of sign language movements, and improve the accuracy and credibility of semantic expression. Summary of the Invention

[0006] In view of the above problems, the purpose of the present invention is to provide a translation method, system and medium for sign language videos to solve the technical problems that the existing sign language translation methods have obvious deficiencies in capturing the complex features of sign language movements, especially the expression of non-linear relationships, resulting in limited accuracy of semantic expression; secondly, most models only support generating natural language text from sign language videos, and lack support for auxiliary outputs such as sign language vocabulary, which limits the multi-scenario applicability of the model; in addition, the existing models fail to fully incorporate the unique needs of sign language translation in structural optimization and loss function design, making it difficult to significantly improve translation performance and robustness.

[0007] In a first aspect, an embodiment of the present application provides a method for translating a sign language video, the method comprising:

[0008] Performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary;

[0009] Using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into a predicted sentence according to the time step;

[0010] The predicted sentence and the sign language vocabulary are used as a translation information file corresponding to the sign language video.

[0011] Optionally, performing deep feedforward encoding on the acquired sign language video by a preset deep feedforward encoding module to obtain sign language video encoding features includes:

[0012] Dividing the acquired sign language video into frames to form continuous video frames;

[0013] Performing feature extraction on the video frame using a preset feature extraction module to obtain video features;

[0014] Performing spatial embedding on the video features by a spatial encoding module in the deep feedforward encoding module to form an embedding vector of fixed dimension;

[0015] adding a positional encoding to the embedding vector to generate an information video vector;

[0016] Performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector;

[0017] The preliminary feature vector is subjected to feature semantic capture by a deep feedforward inference layer in the deep feedforward encoding module to obtain sign language video encoding features.

[0018] Optionally, performing multi-head feature attention calculation on the information video vector through the preset multi-head attention mechanism to obtain a preliminary feature vector includes:

[0019] The multi-head attention mechanism performs multi-head segmentation on the information video vector to obtain multiple subspaces;

[0020] Performing self-attention calculation on the multiple subspaces to obtain similarity data of the information video vectors;

[0021] The feature vectors of the multiple subspaces are fused based on the similarity data to form a preliminary feature vector.

[0022] Optionally, obtaining the predicted sign language vocabulary includes:

[0023] The predicted sign language vocabulary is matched according to the sign language video coding features by the sign language vocabulary architecture in the deep feedforward coding module; wherein,

[0024] The sign language vocabulary framework is a pre-trained sign language vocabulary framework. When training the sign language vocabulary framework, the initial sign language vocabulary framework is repeatedly trained using pre-labeled sign language video samples and vocabulary sequences corresponding to the sign language video samples. During each training process, the vocabulary probability of the sign language video sample to the vocabulary sequence is calculated, and weakly supervised learning is performed on each round of training using the vocabulary probability.

[0025] Optionally, the using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into a predicted sentence according to the time step, includes:

[0026] Performing encoding and decoding processing on the sign language video encoding features and the hidden layer vector related to the encoding and decoding attention mechanism through the encoding and decoding attention mechanism in the deep feedforward decoding module to generate a hidden state vector;

[0027] Performing a nonlinear transformation on the hidden state vector by a deep feedforward inference layer in the deep feedforward decoding module to generate a target hidden state vector;

[0028] Performing a linear transformation through a linear layer in the deep feedforward decoding module to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector;

[0029] Normalizing the linearly transformed vector by a nonlinear function in the deep feedforward decoding module to generate a probability distribution of the mapped current vocabulary;

[0030] Selecting a word with a probability greater than a preset probability from the current word as a predicted word according to the probability distribution;

[0031] The predicted words are connected based on the time step and preset grammatical and semantic rules to generate a predicted sentence.

[0032] Optionally, the depth feedforward encoding module and the depth feedforward decoding module are integrated into a pre-trained depth feedforward model; wherein, when pre-training the depth feedforward model,

[0033] The target loss function of the deep feedforward model is the product of the first weight and the predicted sign language vocabulary loss plus the product of the second weight and the predicted sentence loss; wherein,

[0034] The predicted sign language vocabulary loss is one minus the probability of the deep feedforward model correctly identifying the sign language vocabulary during training;

[0035] The predicted sentence loss is one minus the probability that the deep feedforward model correctly identifies the predicted word from the target language vocabulary in the decoding time step during training.

[0036] Optionally, the deep feedforward inference layer of the deep feedforward model is 2048 layers;

[0037] The deep feedforward inference model has two layers;

[0038] The ratio of the first weight to the second weight is 10:1.

[0039] In a second aspect, an embodiment of the present application provides a sign language video translation system that implements the sign language video translation method described above. The system includes a deep feedforward model; the deep feedforward model includes:

[0040] A deep feedforward encoding module, configured to perform deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predict sign language vocabulary;

[0041] a deep feedforward decoding module, configured to perform deep feedforward inference decoding on the sign language video encoding features to obtain predicted words, and to concatenate the predicted words into a predicted sentence according to a time step;

[0042] An information integration module is used to use the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

[0043] Optionally, the depth feedforward encoding module includes:

[0044] A frame division module is used to divide the acquired sign language video into frames to form continuous video frames;

[0045] A feature extraction module, configured to extract features from the video frame to obtain video features;

[0046] A spatial encoding module, configured to spatially embed the video features to form an embedding vector of fixed dimension;

[0047] a position coding module, configured to add position coding to the embedding vector to generate an information video vector;

[0048] a multi-head attention mechanism for performing multi-head feature attention calculation on the information video vector to obtain a preliminary feature vector;

[0049] A deep feedforward inference layer, configured to perform feature semantic capture on the preliminary feature vector to obtain sign language video coding features;

[0050] The depth feedforward decoding module includes:

[0051] A codec attention mechanism is used to perform codec processing on the sign language video coding features and the hidden layer vector related to the codec attention mechanism to generate a hidden state vector;

[0052] a deep feedforward inference layer, configured to perform a nonlinear transformation on the hidden state vector to generate a target hidden state vector;

[0053] A linear layer, configured to perform a linear transformation to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector;

[0054] a nonlinear function for normalizing the linearly transformed vector to generate a probability distribution of the mapped current vocabulary;

[0055] A word prediction module, configured to select a word with a high probability from the current vocabulary as a predicted word according to the probability distribution;

[0056] The sentence prediction module is used to connect the predicted words based on the time step and preset grammatical and semantic rules to generate a predicted sentence.

[0057] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any optional sign language video translation method described in the first aspect.

[0058] As can be seen from the above technical solutions, the embodiments of the present application provide a method, system, and computer-readable storage medium for translating sign language videos. Compared with the prior art, the present application has the following beneficial effects:

[0059] The deep feedforward model significantly enhances the model's ability to capture complex sign language features, especially the nonlinear relationship expression of sign language movements. It solves the problems of inaccurate semantic expression and omission of detailed information in existing models, and enables the generated text to better retain the semantic details in sign language videos, thereby improving the credibility of translation. The model not only supports the direct generation of natural language text, but also supports the generation of sign language vocabulary and natural language text. It also optimizes the Transformer model structure for sign language translation tasks, adjusts the width of the feedforward layer to 2048, the number of layers of the deep feedforward inference sign language translation model to 2, and the weight of the sign language recognition loss and the sign language translation loss to 10:1, making the optimized deep feedforward model significantly superior to existing technologies in translation indicators (such as BLEU, CHRF, and ROUGE). BRIEF DESCRIPTION OF THE DRAWINGS

[0060] By referring to the following description in conjunction with the accompanying drawings, and with a more complete understanding of the present invention, other objects and results of the present invention will become more apparent and easier to understand. In the accompanying drawings:

[0061] Figure 1 is a flowchart of a method for translating a sign language video according to an embodiment of the present invention;

[0062] Figure 2 Schematic diagram of a deep feedforward encoding module involved in a sign language video translation method according to an embodiment of the present invention;

[0063] Figure 3 Schematic diagram of a deep feedforward decoding module involved in a sign language video translation method according to an embodiment of the present invention;

[0064] Figure 4 1 is an overall schematic diagram of a deep feedforward model involved in a sign language video translation method according to an embodiment of the present invention;

[0065] Figure 5 4 is a logic block diagram of a sign language video translation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The existing sign language translation methods in the art have obvious shortcomings in capturing the complex features of sign language movements, especially the expression of non-linear relationships, resulting in limited accuracy of semantic expression; secondly, most models only support generating natural language text from sign language videos and lack support for auxiliary outputs such as sign language vocabulary, which limits the model's applicability in multiple scenarios; in addition, existing models fail to fully incorporate the unique needs of sign language translation in structural optimization and loss function design, making it difficult to significantly improve translation performance and robustness.

[0067] In response to the above problems, the present invention provides a translation method, system and computer-readable storage medium for sign language videos. First, a preset deep feedforward encoding module is used to perform deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predicted sign language vocabulary. Then, a preset deep feedforward decoding module is used to perform deep feedforward inference decoding on the sign language video encoding features to obtain predicted words, and the predicted words are connected into predicted sentences according to the time step. Then, the predicted sentences and the sign language vocabulary are used as translation information files corresponding to the sign language video. In this way, the deep feedforward encoding module and the deep feedforward decoding module can significantly enhance the model's ability to capture complex sign language features, especially sign language movements. The nonlinear relationship expression solves the problems of inaccurate semantic expression and omission of detailed information in existing models, and can enable the generated text to better retain the semantic details in sign language videos, thereby improving the credibility of translation. The model not only supports the direct generation of natural language text, but also supports the generation of sign language vocabulary and natural language text. In addition, the Transformer model structure is optimized for the sign language translation task, by adjusting the width of the feedforward layer to 2048, the number of layers of the deep feedforward inference sign language translation model to 2, and the weight of the sign language recognition loss and the sign language translation loss to 10:1. As a result, the optimized deep feedforward model significantly outperforms the existing technology in translation indicators (such as BLEU, CHRF, and ROUGE).

[0068] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The description of the following exemplary embodiments is actually merely illustrative and is by no means intended to limit the present invention and its application or use. Technologies and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, the technologies and equipment should be considered part of the specification. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0069] Figure 1 A flowchart of a sign language video translation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0070] S1: Perform deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary;

[0071] S2: Using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into a predicted sentence according to the time step;

[0072] S3: Using the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

[0073] Step S1 is a process of performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain the sign language video encoding features and predict the sign language vocabulary. The deep feedforward encoding module used in this step is as follows: Figure 2 As shown, the acquired sign language video is subjected to deep feedforward encoding by a preset deep feedforward encoding module to obtain sign language video encoding features, including:

[0074] S11: dividing the acquired sign language video into frames to form continuous video frames;

[0075] S12: extracting features from the video frame using a preset feature extraction module to obtain video features;

[0076] S13: spatially embedding the video features through the spatial encoding module in the deep feedforward encoding module to form an embedding vector of fixed dimension;

[0077] S14: adding a position code to the embedding vector to generate an information video vector;

[0078] S15: performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector;

[0079] S16: Perform feature semantic capture on the preliminary feature vector through the deep feedforward inference layer in the deep feedforward encoding module to obtain sign language video encoding features.

[0080] In a more specific embodiment,

[0081] Performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector, including:

[0082] S151: performing multi-head segmentation on the information video vector by the multi-head attention mechanism to obtain multiple subspaces;

[0083] S152: performing self-attention calculation on the multiple subspaces to obtain similarity data of the information video vector;

[0084] S153: Fusing the feature vectors of the multiple subspaces based on the similarity data to form a preliminary feature vector.

[0085] Step (1): After obtaining the video frame and extracting the features of the video frame, first The video features are spatially embedded and converted into an embedding vector of fixed dimension , for subsequent processing; the spatial embedding operation can capture the spatial features of the image, enabling the model to better understand the visual information in the video frame. The fixed-dimensional embedding vector can be expressed as follows:

[0086] ;

[0087] Step (2): Embedding vector in fixed dimension Add position encoding , generate a new feature vector , which can be called the information video vector; among them, position encoding is performed after feature extraction and spatial embedding are completed and before the feature vector is input into the multi-head self-attention layer to preserve the temporal order information in the sequence; the information video vector and are expressed as follows:

[0088]

[0089] Step (3): Through the preset multi-head attention mechanism Information video vector Processing to generate preliminary feature vector Among them, the multi-head attention mechanism allows the model to focus on different information in different representation subspaces, thereby capturing more complex dependencies, layer normalization This mechanism is used to accelerate training and avoid the vanishing gradient problem. Specifically, the information it focuses on includes semantic features, temporal dependencies, and spatial relationships. Semantic features involve gesture shape, size, and orientation. Temporal dependencies include changes in motion between adjacent frames, as well as motion sequences and semantic coherence across multiple time steps. Spatial relationships involve the relative positions of gestures and the spatial arrangement of gestures with other body parts. To capture this information, visual features are first extracted from each video frame and converted into a vector form suitable for Transformer processing through spatial embedding. Positional encoding is applied to capture the temporal order of different elements in the sequence, providing temporal context for the multi-head self-attention mechanism. Furthermore, in the multi-head self-attention layer, the feature vector is split into multiple heads, each of which independently performs self-attention calculations in a different representation subspace. By calculating the similarity between feature vectors, the model is able to capture different dependencies and feature information in each subspace. Finally, the feature vectors processed by the multi-head self-attention mechanism are fused to generate a preliminary feature vector representation containing rich information:

[0090]

[0091] Step (4): The deep feedforward inference layer in the deep feedforward encoding module proposed in this embodiment performs feature semantic capture on the preliminary feature vector to obtain the final sign language video encoding features. .

[0092]

[0093] in, The deep feedforward inference layer proposed above is represented. After repeated experiments, it is concluded that the deep feedforward inference layer proposed in this embodiment enables the sign language translation model to capture more complex sign language feature information and stronger semantic expression accuracy. The formula of the deep feedforward inference layer is expressed as:

[0094]

[0095] in, is the input video frame, and is the learnable weight matrix, and is the learnable bias. Symbol and Represents an activation function and a learning parameter respectively, and we find The optimal value is 0.01. Comparative experiments were conducted with different depth feedforward inference layer widths, setting the depth feedforward inference layer widths to 512, 1024, 2048, 3072, 4096, and 5120. The experiments showed that when the depth feedforward inference layer width is 2048, the model performance is optimal.

[0096] In this embodiment, the process of obtaining the predicted sign language vocabulary includes:

[0097] S16: Matching predicted sign language vocabulary according to the sign language video coding features through the sign language vocabulary architecture in the deep feedforward coding module; wherein,

[0098] The sign language vocabulary framework is a pre-trained sign language vocabulary framework (the model or framework applied by the entire method is trained simultaneously). When training the sign language vocabulary framework, the initial sign language vocabulary framework is repeatedly trained using pre-labeled sign language video samples and vocabulary sequences corresponding to the sign language video samples. During each training process, the vocabulary probability of the sign language video sample to the vocabulary sequence is calculated, and the vocabulary probability is used to perform weakly supervised learning on each round of training.

[0099] That is, in this specific embodiment, in the process of training the model used in this method, we not only focus on how to Extract features from the , and map these features to the corresponding predicted sign language vocabulary (sign language vocabulary sequence) Sign language vocabulary sequence It is obtained by annotating a large number of sign language videos, and it contains the vocabulary information corresponding to each sign language action in the video. During the training phase, the model will be exposed to these annotated sign language videos and their corresponding sign language vocabulary sequences, and learn how to transform the video frame sequence into the sign language vocabulary sequence by optimizing the connection temporal classification (CTC) loss function. Mapping to sign language vocabulary sequences The CTC loss function allows the model to learn this mapping relationship without the need for precise alignment of each video frame, thereby achieving weakly supervised learning. The following formula is expressed as the calculation of the sign language video frame To sign language vocabulary sequence Probability , weakly supervised learning is achieved through the connection temporal classification (CTC) loss function:

[0100]

[0101] in, represents a possible path from a sign language video frame to a sign language vocabulary, Represents the set of all possible paths.

[0102] Step S2 is a process of using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps. The process of using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps includes:

[0103] S21: performing encoding and decoding processing on the sign language video encoding features and the hidden layer vector related to the encoding and decoding attention mechanism through the encoding and decoding attention mechanism in the deep feedforward decoding module to generate a hidden state vector;

[0104] S22: Performing a nonlinear transformation on the hidden state vector through a deep feedforward inference layer in the deep feedforward decoding module to generate a target hidden state vector;

[0105] S23: performing a linear transformation through a linear layer in the deep feedforward decoding module to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector;

[0106] S24: performing normalization processing on the linearly transformed vector by a nonlinear function in the deep feedforward decoding module to generate a probability distribution of the mapped current vocabulary;

[0107] S25: selecting a word with a probability greater than a preset probability from the current word as a predicted word according to the probability distribution;

[0108] S26: Connecting the predicted words based on the time step and preset grammatical and semantic rules to generate a predicted sentence.

[0109] Specifically, in this embodiment, the function of the module is to decode the video coding features obtained in the first step into the corresponding translation text. More specifically, step (1): the sign language video coding features obtained above are decoded into the corresponding translation text. Send to the attached Figure 3 The deep feedforward decoding module shown in Figure 3 As shown. Through the encoder-decoder multi-head attention mechanism in the deep feedforward decoding module For the hidden layer vector and the video feature vector generated by the encoder Processing to generate a new hidden state vector :

[0110] ;

[0111] The hidden layer vector It is gradually constructed and updated during the processing of the self-attention and encoder-decoder attention mechanisms of the decoder module, and it integrates word embedding information and encoded feature information.

[0112] Step (2): Through the deep feedforward inference layer For the hidden state vector Perform nonlinear transformation to generate the final hidden state vector , referred to as the target hidden state vector in this embodiment;

[0113] ;

[0114] Step (3): Target hidden state vector obtained from step (2) To predict words The probability of . First, the hidden state vector The linear transformation is performed through the linear layer to form a linear transformation vector, that is, the feature is mapped to the dimension of the vocabulary, and then the transformed vector is normalized by the SoftMax function to obtain the predicted probability distribution of each word at the current time step. The SoftMax function ensures that the sum of all probabilities is 1, so it can be interpreted as a probability distribution. According to this probability distribution, the word with the largest probability is selected as the predicted word at the current time step. In the decoding phase, the model generates words one by one and connects the words predicted at each time step in sequence to form a complete sentence. This process needs to consider the grammatical and semantic coherence between words, and the decoding process usually stops when a specific end marker (EOS, indicating the end of the sentence) is encountered. At this time, the model has generated a complete sentence. In this specific implementation, the maximum likelihood estimation is used to obtain the word count from the sign language video frame. To the translated sentence The conditional probability of , the calculation formula is as follows:

[0115]

[0116] It should be noted that the depth feedforward encoding module and the depth feedforward decoding module are integrated into a pre-trained depth feedforward model; wherein, when pre-training the depth feedforward model,

[0117] The target loss function of the deep feedforward model is the product of the first weight and the predicted sign language vocabulary loss plus the product of the second weight and the predicted sentence loss; wherein,

[0118] The predicted sign language vocabulary loss is one minus the probability of the deep feedforward model correctly identifying the sign language vocabulary during training;

[0119] The predicted sentence loss is one minus the probability that the deep feedforward model correctly identifies the predicted word from the target language vocabulary in the decoding time step during training.

[0120] Specifically, is the total objective loss function, which aims to adjust and The weight ratio enables the model to simultaneously translate sign language videos into sign language vocabulary and sentences. Training stops when the loss function no longer decreases significantly over multiple consecutive epochs.

[0121]

[0122] in, The weight value of the recognition loss function is the weight when testing the sign language vocabulary, which is called the first weight. is the weight value of the translation loss function, that is, the weight when predicting the sentence, called the second weight.

[0123] This is the loss function for sign language recognition, that is, the loss function used by the model to predict sign language vocabulary during training. This loss function aims to measure the performance of the model in the sign language recognition task and minimize recognition errors.

[0124]

[0125] in, Indicates correct recognition of sign language vocabulary sequence probability.

[0126] It is the translation loss function, which is the loss generated when generating the predicted sentence. This loss function is designed to measure the performance of the model in the sign language translation task and minimize translation errors.

[0127]

[0128] in, Denotes the predicted word at decoding time step u The probability of Indicates the size of the target language vocabulary.

[0129] After repeated experiments, comparative experiments were conducted using different widths of deep feedforward inference layers, and the widths of the deep feedforward inference layers were set to 512, 1024, 2048, 3072, 4096, and 5120, respectively. Experiments have shown that when the width of the deep feedforward inference layer is 2048, the model performance is optimal. Comparative experiments were conducted using different numbers of layers of deep feedforward inference models, and the numbers of layers of the deep feedforward inference models were set to 1, 2, 3, 4, 5, and 6, respectively. Experiments have shown that when the number of layers of the deep feedforward inference model is 2, the model obtains better nonlinear expression capabilities and deep reasoning capabilities. Most existing sign language translation models use a single end-to-end translation architecture, which only supports direct generation of natural language text from sign language videos. This embodiment can not only directly translate sign language videos into text end-to-end, but also translate them into sign language vocabulary and translated text. And two loss functions are used: recognition loss function (Formula below) and translation loss function (Formula below) , and thus a comparative experiment was conducted by using different weight ratios of sign language recognition loss function and sign language translation loss function. The sign language translation loss value was fixed at 1, and the sign language recognition loss values were 0.5, 1, 2.5, 5, 10, and 20. The experiment proved that when the weight of sign language recognition loss and sign language translation loss was 10:1, the overall performance of the model was optimal. Therefore, in this embodiment, the deep feedforward inference layer of the deep feedforward model is 2048 layers; the number of layers of the deep feedforward inference model is two layers; the ratio of the first weight to the second weight is 10:1, so as to produce a more accurate prediction effect, which can greatly improve the translation indicators of BLEU, CHRF, and ROUGE, and improve the sign language prediction performance.

[0130] The meanings of the symbols in the above formula are as follows:

[0131]

[0132]

[0133]

[0134] As described above, the sign language video translation method provided by the present invention can significantly enhance the model's ability to capture complex sign language features, especially the nonlinear relationship expression of sign language movements, through the deep feedforward encoding module and the deep feedforward decoding module, solve the problems of inaccurate semantic expression and omission of detailed information in the existing model, and enable the generated text to better retain the semantic details in the sign language video, thereby improving the credibility of the translation; the model not only supports the direct generation of natural language text, but also supports the generation of sign language vocabulary and natural language text; and optimizes the Transformer model structure for the sign language translation task, adjusts the width of the feedforward layer to 2048, the number of layers of the deep feedforward inference sign language translation model to 2 layers, and the weight of the sign language recognition loss and the sign language translation loss to 10:1, so that the optimized deep feedforward model is significantly superior to the existing technology in translation indicators (such as BLEU, CHRF, ROUGE).

[0135] The following introduces a sign language video translation system provided in an embodiment of the present application. The sign language video translation system described below and the sign language video translation method described above can refer to each other.

[0136] like Figure 4 As shown, the present invention also provides a sign language video translation system 100, which can implement the sign language video translation method as described above. The system includes a deep feedforward model; the deep feedforward model includes:

[0137] A deep feedforward encoding module 101 is configured to perform deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predict sign language vocabulary;

[0138] A deep feedforward decoding module 102 is configured to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and to concatenate the predicted words into a predicted sentence according to a time step;

[0139] The information integration module 103 is configured to use the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

[0140] In this embodiment, the depth feedforward encoding module 101 includes:

[0141] A frame division module 1011 is used to divide the acquired sign language video into frames to form continuous video frames;

[0142] A feature extraction module 1012 is configured to extract features from the video frame to obtain video features;

[0143] A spatial encoding module 1013, configured to perform spatial embedding on the video features to form an embedding vector of fixed dimension;

[0144] a position coding module 1014 for adding position coding to the embedding vector to generate an information video vector;

[0145] a multi-head attention mechanism 1015 for performing multi-head feature attention calculation on the information video vector to obtain a preliminary feature vector;

[0146] A deep feedforward inference layer 1016 is used to perform feature semantic capture on the preliminary feature vector to obtain sign language video coding features;

[0147] The depth feedforward decoding module includes:

[0148] A codec attention mechanism 1021 is configured to perform codec processing on the sign language video coding features and the hidden layer vector associated with the codec attention mechanism to generate a hidden state vector;

[0149] A deep feedforward inference layer 1022, configured to perform a nonlinear transformation on the hidden state vector to generate a target hidden state vector;

[0150] A linear layer 1023, configured to perform a linear transformation to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector;

[0151] A nonlinear function 1024 is used to normalize the linearly transformed vector to generate a probability distribution of the mapped current vocabulary;

[0152] A word prediction module 1025 is configured to select a word with a high probability from the current vocabulary as a predicted word according to the probability distribution;

[0153] The sentence prediction module 1026 is configured to connect the predicted words based on the time step and preset grammatical and semantic rules to generate a predicted sentence.

[0154] The sign language video translation system 100 provided by the present invention generally performs calculations as described in the aforementioned sign language video translation method. For more specific implementation processes, reference may be made to the specific embodiments of the aforementioned sign language video translation method.

[0155] As described above, the sign language video translation system 100 provided by the present invention, such as the sign language video translation method described above, can also solve the problems of inaccurate semantic expression and omission of detailed information in existing models, and can enable the generated text to better retain the semantic details in the sign language video, thereby improving the credibility of the translation, improving the translation indicators, and improving the translation effect.

[0156] An embodiment of the present invention further provides a computer-readable storage medium (not shown in the figure). The storage medium may be non-volatile or volatile, and the storage medium stores a computer program. When the computer program is executed by a processor, the following is achieved:

[0157] Performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary;

[0158] Using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into a predicted sentence according to the time step;

[0159] The predicted sentence and the sign language vocabulary are used as a translation information file corresponding to the sign language video.

[0160] Specifically, the specific implementation method when the computer program is executed by the processor can refer to the description of the relevant steps in the translation method of sign language video in the embodiment, which will not be repeated here.

[0161] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0162] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0163] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0164] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0165] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0166] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0167] It should be noted that the various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the methods, devices, electronic devices, and computer-readable storage media are described in a relatively simple manner because they are basically similar to the method embodiments. For relevant parts, refer to the partial description of the method embodiments. The methods, devices, electronic devices, and media described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0168] The above description of the sign language video translation method, system, and storage medium proposed in accordance with the present invention is described by way of example with reference to the accompanying drawings. However, those skilled in the art will appreciate that various improvements may be made to the sign language video translation method, system, and storage medium proposed in accordance with the present invention without departing from the scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the contents of the appended claims.

Claims

1. A method for translating a sign language video, characterized in that: include: Performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary; A preset deep feedforward decoding module is used to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and the predicted words are connected into predicted sentences according to the time step; wherein, it includes: encoding and decoding the sign language video coding features and the hidden layer vectors related to the codec attention mechanism through the codec attention mechanism in the deep feedforward decoding module to generate a hidden state vector; performing a nonlinear transformation on the hidden state vector through the deep feedforward inference layer in the deep feedforward decoding module to generate a target hidden state vector; performing a linear transformation through the linear layer in the deep feedforward decoding module to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; normalizing the linearly transformed vector through the nonlinear function in the deep feedforward decoding module to generate a probability distribution of the mapped current vocabulary; selecting a vocabulary with a probability greater than a preset probability in the current vocabulary according to the probability distribution as a predicted word; and connecting the predicted words based on the time step and preset grammatical and semantic rules to generate a predicted sentence; The deep feedforward encoding module and the deep feedforward decoding module are integrated into a pre-trained deep feedforward model; wherein, when pre-training the deep feedforward model, the target loss function of the deep feedforward model is the product of a first weight and a predicted sign language vocabulary loss plus a second weight and a predicted sentence loss; wherein the predicted sign language vocabulary loss is one minus the probability that the deep feedforward model correctly recognizes the sign language vocabulary during training; and the predicted sentence loss is one minus the probability that the deep feedforward model correctly confirms the predicted word from the target language vocabulary in a decoding time step during training; The predicted sentence and the sign language vocabulary are used as a translation information file corresponding to the sign language video.

2. The method for translating a sign language video according to claim 1, wherein: The step of performing deep feedforward encoding on the acquired sign language video by a preset deep feedforward encoding module to obtain sign language video encoding features includes: Dividing the acquired sign language video into frames to form continuous video frames; Performing feature extraction on the video frame using a preset feature extraction module to obtain video features; Performing spatial embedding on the video features by a spatial encoding module in the deep feedforward encoding module to form an embedding vector of fixed dimension; adding a positional encoding to the embedding vector to generate an information video vector; Performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector; The preliminary feature vector is subjected to feature semantic capture by a deep feedforward inference layer in the deep feedforward encoding module to obtain sign language video encoding features.

3. The method for translating a sign language video according to claim 2, wherein: The performing multi-head feature attention calculation on the information video vector by a preset multi-head attention mechanism to obtain a preliminary feature vector includes: Performing multi-head segmentation on the information video vector by the multi-head attention mechanism to obtain multiple subspaces; Performing self-attention calculation on the multiple subspaces to obtain similarity data of the information video vectors; The feature vectors of the multiple subspaces are fused based on the similarity data to form a preliminary feature vector.

4. The method for translating a sign language video according to claim 3, wherein: Obtaining the predicted sign language vocabulary includes: The predicted sign language vocabulary is matched according to the sign language video coding features by the sign language vocabulary architecture in the deep feedforward coding module; wherein, The sign language vocabulary framework is a pre-trained sign language vocabulary framework. When training the sign language vocabulary framework, the initial sign language vocabulary framework is repeatedly trained using pre-labeled sign language video samples and vocabulary sequences corresponding to the sign language video samples. During each training process, the vocabulary probability of the sign language video sample to the vocabulary sequence is calculated, and weakly supervised learning is performed on each round of training using the vocabulary probability.

5. The method for translating a sign language video according to claim 4, wherein: The deep feedforward inference layer of the deep feedforward model is 2048 layers; The number of layers of the deep feedforward model is two; The ratio of the first weight to the second weight is 10:

1.

6. A sign language video translation system, characterized in that: A method for translating a sign language video according to any one of claims 1 to 5 is implemented, wherein the system includes a deep feedforward model; the deep feedforward model includes: A deep feedforward encoding module is used to perform deep feedforward encoding on the acquired sign language video to obtain sign language video coding features and predict sign language vocabulary; wherein the deep feedforward encoding module includes: A frame division module is used to divide the acquired sign language video into frames to form continuous video frames; A feature extraction module, configured to extract features from the video frame to obtain video features; A spatial encoding module, configured to spatially embed the video features to form an embedding vector of fixed dimension; a position coding module, configured to add position coding to the embedding vector to generate an information video vector; a multi-head attention mechanism for performing multi-head feature attention calculation on the information video vector to obtain a preliminary feature vector; A deep feedforward inference layer, configured to perform feature semantic capture on the preliminary feature vector to obtain sign language video coding features; The depth feedforward decoding module includes: A codec attention mechanism is used to perform codec processing on the sign language video coding features and the hidden layer vector related to the codec attention mechanism to generate a hidden state vector; a deep feedforward inference layer, configured to perform a nonlinear transformation on the hidden state vector to generate a target hidden state vector; A linear layer, configured to perform a linear transformation to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; a nonlinear function for normalizing the linearly transformed vector to generate a probability distribution of the mapped current vocabulary; A word prediction module, configured to select a word with a high probability from the current vocabulary as a predicted word according to the probability distribution; A sentence prediction module is used to connect the predicted words based on the time step and preset grammatical and semantic rules to generate a predicted sentence; a deep feedforward decoding module is used to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connect the predicted words into a predicted sentence based on the time step; An information integration module is used to use the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for translating a sign language video according to any one of claims 1 to 5 is implemented.

Images