Sign language video translation method and system and medium

Abstract

The invention provides a sign language video translation method and system and a computer readable storage medium, and the method comprises the steps: carrying out the deep feed-forward coding of an obtained sign language video through a preset deep feed-forward coding module, so as to obtain sign language video coding features and prediction sign language vocabularies; a preset depth feed-forward decoding module is adopted to carry out depth feed-forward inference decoding on the sign language video coding features to obtain prediction words, the prediction words are connected into prediction statements according to time steps, and then the prediction statements and the sign language vocabularies serve as translation information files corresponding to the sign language video. By means of the method, the capturing capacity of the model for complex sign language features, especially non-linear relation expression of sign language actions, can be remarkably enhanced, the problems that an existing model is inaccurate in semantic expression and missing in detailed information are solved, the generated text can better reserve semantic details in sign language videos, and the reliability and accuracy of translation are improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language technology, and more specifically to a self-translation method of sign language, and more specifically to a translation method, system and medium for sign language video. Background Art

[0002] Sign language is a unique visual language used by the hearing-impaired that facilitates visual communication by using hand information and non-hand information. A perfect translation model for sign language can convert visual language into natural language without additional steps. Sign language is a brand new language with unique language logic, grammar, and semantic structure. Semantic segmentation and sign language understanding are two problems encountered when translating sign language directly into natural language text. In the early and mid-stage fields of artificial intelligence, more research focused on sign language recognition tasks, including isolated and sequential sign language recognition, rather than fully translating sign language videos into the target language. Sign language recognition recognizes sign language videos as text and is the written form of sign language.

[0003] The goal of sign language translation is to achieve the conversion of sign language video into natural language text, thereby building a bridge for communication between the hearing-impaired and society.

[0004] However, existing sign language translation methods have obvious shortcomings in capturing the complex features of sign language movements, especially the expression of non-linear relationships, resulting in limited accuracy of semantic expression; secondly, most models only support generating natural language text from sign language videos, and lack support for auxiliary outputs such as sign language vocabulary, which limits the model's applicability in multiple scenarios; in addition, existing models fail to fully incorporate the unique needs of sign language translation in terms of structural optimization and loss function design, making it difficult to significantly improve translation performance and robustness.

[0005] Therefore, there is an urgent need for a translation method, system and medium for sign language videos that can simultaneously generate sign language vocabulary and sentences, enhance the ability to capture complex features of sign language movements, and improve the accuracy and credibility of semantic expression. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a translation method, system and medium for sign language video to solve the problem that the existing sign language translation methods are not good at capturing the complex features of sign language movements, especially the expression of non-linear relationships. Traditional methods show obvious deficiencies, resulting in limited accuracy of semantic expression; secondly, most models only support the generation of natural language text from sign language videos, and lack support for auxiliary outputs such as sign language vocabulary, which limits the multi-scenario applicability of the model; in addition, the existing models fail to fully combine the unique needs of sign language translation in structural optimization and loss function design, and it is difficult to significantly improve the translation performance and robustness.

[0007] In a first aspect, an embodiment of the present application provides a method for translating a sign language video, the method for translating a sign language video comprising: Performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary; Using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps; The predicted sentence and the sign language vocabulary are used as a translation information file corresponding to the sign language video.

[0008] Optionally, performing deep feedforward encoding on the acquired sign language video by a preset deep feedforward encoding module to acquire a sign language video encoding feature includes: Dividing the acquired sign language video into frames to form continuous video frames; Extracting features from the video frame using a preset feature extraction module to obtain video features; The video features are spatially embedded by a spatial encoding module in the deep feedforward encoding module to form an embedding vector of fixed dimension; adding a positional encoding to the embedding vector to generate an information video vector; Performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector; The preliminary feature vector is semantically captured through a deep feedforward inference layer in the deep feedforward encoding module to obtain sign language video encoding features.

[0009] Optionally, performing multi-head feature attention calculation on the information video vector through the preset multi-head attention mechanism to obtain a preliminary feature vector includes: The multi-head attention mechanism performs multi-head segmentation on the information video vector to obtain multiple subspaces; Performing self-attention calculation on the multiple subspaces to obtain similarity data of the information video vectors; The feature vectors of the multiple subspaces are fused based on the similarity data to form a preliminary feature vector.

[0010] Optionally, acquiring the predicted sign language vocabulary includes: The predicted sign language vocabulary is matched according to the sign language video coding features through the sign language vocabulary architecture in the deep feedforward coding module; wherein, The sign language vocabulary framework is a pre-trained sign language vocabulary framework. When training the sign language vocabulary framework, the initial sign language vocabulary framework is repeatedly trained using pre-labeled sign language video samples and vocabulary sequences corresponding to the sign language video samples, and the vocabulary probability of the sign language video samples to the vocabulary sequence is calculated during each training process, and weakly supervised learning is performed on each round of training using the vocabulary probability.

[0011] Optionally, the using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into a predicted sentence according to the time step, includes: Performing encoding and decoding processing on the sign language video encoding features and the hidden layer vector related to the encoding and decoding attention mechanism through the encoding and decoding attention mechanism in the deep feedforward decoding module to generate a hidden state vector; Performing a nonlinear transformation on the hidden state vector through a deep feedforward inference layer in the deep feedforward decoding module to generate a target hidden state vector; Performing a linear transformation through a linear layer in the deep feedforward decoding module to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; Normalizing the linearly transformed vector by a nonlinear function in the deep feedforward decoding module to generate a probability distribution of the mapped current vocabulary; Selecting a word with a probability greater than a preset probability from the current word as a predicted word according to the probability distribution; The predicted words are connected based on the time step and preset grammatical semantic rules to generate a predicted sentence.

[0012] Optionally, the deep feedforward encoding module and the deep feedforward decoding module are integrated in a pre-trained deep feedforward model; wherein, when pre-training the deep feedforward model, The target loss function of the deep feedforward model is the product of the first weight and the predicted sign language vocabulary loss plus the product of the second weight and the predicted sentence loss; wherein, The predicted sign language vocabulary loss is one minus the probability of the deep feedforward model correctly identifying the sign language vocabulary during training; The prediction sentence loss is one minus the probability that the deep feedforward model correctly confirms the predicted word from the target language vocabulary in the decoding time step during training.

[0013] Optionally, the deep feedforward inference layer of the deep feedforward model is 2048 layers; The deep feedforward inference model has two layers; The ratio of the first weight to the second weight is 10:1.

[0014] In a second aspect, an embodiment of the present application provides a translation system for a sign language video, which implements the translation method for a sign language video as described above, and the system includes a deep feedforward model; the deep feedforward model includes: A deep feedforward encoding module, used for performing deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predict sign language vocabulary; A deep feedforward decoding module, used for performing deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps; The information integration module is used to use the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

[0015] Optionally, the depth feedforward encoding module includes: A frame division module, used for dividing the acquired sign language video into frames to form continuous video frames; A feature extraction module, used for extracting features from the video frame to obtain video features; A spatial encoding module, used for spatially embedding the video features to form an embedding vector of fixed dimension; A position coding module, used for adding position coding to the embedding vector to generate an information video vector; A multi-head attention mechanism, for performing multi-head feature attention calculation on the information video vector to obtain a preliminary feature vector; A deep feed-forward inference layer, used for performing feature semantic capture on the preliminary feature vector to obtain sign language video coding features; The depth feedforward decoding module comprises: A codec attention mechanism is used to perform codec processing on the sign language video coding features and the hidden layer vector related to the codec attention mechanism to generate a hidden state vector; A deep feedforward inference layer, configured to perform a nonlinear transformation on the hidden state vector to generate a target hidden state vector; A linear layer, configured to perform a linear transformation to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; A nonlinear function, used for normalizing the linearly transformed vector to generate a probability distribution of the mapped current vocabulary; A word prediction module, used for selecting a word with a high probability from the current vocabulary as a predicted word according to the probability distribution; The sentence prediction module is used to connect the predicted words based on the time step and preset grammatical semantic rules to generate a predicted sentence.

[0016] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any optional method for translating a sign language video according to the first aspect.

[0017] It can be seen from the above technical solutions that the embodiments of the present application provide a method, system and computer-readable storage medium for translating a sign language video. Compared with the prior art, the present application has the following beneficial effects: The deep feedforward model significantly enhances the model's ability to capture complex sign language features, especially the nonlinear relationship expression of sign language movements, and solves the problems of inaccurate semantic expression and missing detailed information in existing models. It enables the generated text to better retain the semantic details in the sign language video and improve the credibility of the translation. The model not only supports the direct generation of natural language text, but also supports the generation of sign language vocabulary and natural language text. It also optimizes the Transformer model structure for the sign language translation task, adjusts the width of the feedforward layer to 2048, the number of layers of the deep feedforward reasoning sign language translation model to 2 layers, and the weight of the sign language recognition loss and the sign language translation loss to 10:1, so that the optimized deep feedforward model is significantly better than the existing technology in translation indicators (such as BLEU, CHRF, ROUGE). BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By referring to the following description in conjunction with the accompanying drawings, and with a more comprehensive understanding of the present invention, other objects and results of the present invention will become more clear and easy to understand. In the accompanying drawings: Figure 1 is a flowchart of a method for translating a sign language video according to an embodiment of the present invention; Figure 2 A schematic diagram of a deep feedforward encoding module involved in a method for translating a sign language video according to an embodiment of the present invention; Figure 3 A schematic diagram of a deep feedforward decoding module involved in a method for translating a sign language video according to an embodiment of the present invention; Figure 4 is an overall schematic diagram of a deep feedforward model involved in a method for translating a sign language video according to an embodiment of the present invention; Figure 5 The figure is a logic block diagram of a sign language video translation system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The existing sign language translation methods in the art have obvious shortcomings in capturing the complex features of sign language movements, especially the expression of non-linear relationships, resulting in limited accuracy of semantic expression; secondly, most models only support generating natural language text from sign language videos, and lack support for auxiliary outputs such as sign language vocabulary, which limits the model's applicability in multiple scenarios; in addition, existing models fail to fully incorporate the unique needs of sign language translation in terms of structural optimization and loss function design, making it difficult to significantly improve translation performance and robustness.

[0020] In view of the above problems, the present invention provides a translation method, system and computer-readable storage medium for a sign language video. First, a preset deep feedforward encoding module is used to perform deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predicted sign language vocabulary. Then, a preset deep feedforward decoding module is used to perform deep feedforward inference decoding on the sign language video encoding features to obtain predicted words, and the predicted words are connected into predicted sentences according to the time step. Then, the predicted sentences and the sign language vocabulary are used as translation information files corresponding to the sign language video. In this way, the deep feedforward encoding module and the deep feedforward decoding module can significantly enhance the model's ability to capture complex sign language features, especially sign language movements. The nonlinear relationship expression solves the problems of inaccurate semantic expression and missing detailed information in the existing model, and can enable the generated text to better retain the semantic details in the sign language video and improve the credibility of the translation. The model not only supports the direct generation of natural language text, but also supports the generation of sign language vocabulary and natural language text. In addition, the Transformer model structure is optimized for the sign language translation task, the width of the feedforward layer is adjusted to 2048, the number of layers of the deep feedforward reasoning sign language translation model is adjusted to 2 layers, and the weight of the sign language recognition loss and the sign language translation loss is 10:1, so that the optimized deep feedforward model is significantly better than the existing technology in translation indicators (such as BLEU, CHRF, ROUGE).

[0021] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The description of the following exemplary embodiments is actually merely illustrative and is by no means intended to limit the present invention and its application or use. The techniques and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, the techniques and equipment should be considered as part of the specification. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0022] Figure 1A flowchart of a method for translating a sign language video provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes: S1: performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary; S2: using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps; S3: Using the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

[0023] Step S1 is a process of performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain the sign language video encoding features and predict the sign language vocabulary. The deep feedforward encoding module used in this step is as follows: Figure 2 As shown, the acquired sign language video is deeply feedforward encoded by a preset deep feedforward encoding module to obtain the sign language video encoding features, including: S11: dividing the acquired sign language video into frames to form continuous video frames; S12: extracting features from the video frame using a preset feature extraction module to obtain video features; S13: spatially embedding the video features through the spatial encoding module in the deep feedforward encoding module to form an embedding vector of fixed dimension; S14: adding a position code to the embedding vector to generate an information video vector; S15: performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector; S16: Perform feature semantic capture on the preliminary feature vector through the deep feedforward inference layer in the deep feedforward encoding module to obtain sign language video encoding features.

[0024] In a more specific embodiment, Performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector, including: S151: performing multi-head segmentation on the information video vector through the multi-head attention mechanism to obtain multiple subspaces; S152: performing self-attention calculation on the multiple subspaces to obtain similarity data of the information video vector; S153: Based on the similarity data, the feature vectors of the multiple subspaces are merged to form a preliminary feature vector.

[0025] Step (1): After obtaining the video frame and performing feature extraction on the video frame, first perform feature extraction on the input video frame. The video features are spatially embedded and converted into an embedding vector of fixed dimension. , for subsequent processing; the spatial embedding operation can capture the spatial features of the image, so that the model can better understand the visual information in the video frame. The fixed-dimensional embedding vector can be expressed as follows: ; Step (2): Embedding vector in fixed dimension Add position encoding , generate a new feature vector , which can be called an information video vector; where position encoding is performed after feature extraction and spatial embedding are completed and before the feature vector is input into the multi-head self-attention layer, which is used to retain the time order information in the sequence; the information video vector and are expressed as follows:

[0026] Step (3): Through the preset multi-head attention mechanism Information video vector Processing to generate preliminary feature vector ; Among them, the multi-head attention mechanism allows the model to focus on different information in different representation subspaces, thereby capturing more complex dependencies, layer normalization It is used to accelerate the training process and avoid the problem of gradient disappearance. The information that this mechanism focuses on specifically includes semantic features, temporal dependencies, and spatial relationships. Semantic features involve the shape, size, direction, etc. of gestures. Temporal dependencies include action changes between adjacent frames, as well as action sequences and semantic coherence across multiple time steps. Spatial relationships involve the relative positions between gestures and the spatial layout of gestures and other parts of the body. In order to obtain this information, visual features are first extracted from each video frame, and these features are converted into vector forms suitable for Transformer processing through spatial embedding. Position encoding is applied to capture the temporal order information of different elements in the sequence, providing the context of the time dimension for the multi-head self-attention mechanism; and in the multi-head self-attention layer, the feature vector is divided into multiple heads, each of which performs self-attention calculations independently in different representation subspaces. By calculating the similarity between feature vectors, the model can capture different dependencies and feature information in each subspace. Finally, the feature vectors processed by the multi-head self-attention mechanism are fused to generate a preliminary feature vector representation containing rich information:

[0027] Step (4): The deep feedforward inference layer in the deep feedforward encoding module proposed in this embodiment performs feature semantic capture on the preliminary feature vector to obtain the final sign language video encoding feature .

[0028]

[0029] in, Representing the deep feedforward inference layer proposed above, after repeated experiments, it is concluded that the deep feedforward inference layer proposed in this embodiment enables the sign language translation model to capture more complex sign language feature information and stronger semantic expression accuracy; the formula of the deep feedforward inference layer is expressed as:

[0030] in, is the input video frame, and is the learnable weight matrix, and is the learnable bias. Symbol and Represents an activation function and a learning parameter respectively. We find The optimal value of is 0.01. Through comparative experiments with different depth feedforward inference layer widths, the depth feedforward inference layer widths are set to 512, 1024, 2048, 3072, 4096, and 5120 respectively. The experiment proves that when the depth feedforward inference layer width is 2048, the model performance is optimal.

[0031] In this embodiment, the process of obtaining the predicted sign language vocabulary includes: S16: Matching predicted sign language vocabulary according to the sign language video coding features through the sign language vocabulary architecture in the deep feedforward coding module; wherein, The sign language vocabulary framework is a pre-trained sign language vocabulary framework (the model or framework to which the entire method is applied is trained simultaneously). When training the sign language vocabulary framework, the initial sign language vocabulary framework is repeatedly trained using pre-labeled sign language video samples and vocabulary sequences corresponding to the sign language video samples, and the vocabulary probability of the sign language video samples to the vocabulary sequence is calculated during each training process, and weakly supervised learning is performed on each round of training using the vocabulary probability.

[0032] That is, in this specific embodiment, in the process of training the model used in this method, we not only focus on how to Extract features from the , and map these features to the corresponding predicted sign language vocabulary (sign language vocabulary sequence) Previous: Sign language vocabulary sequence It is obtained by annotating a large number of sign language videos, and it contains the vocabulary information corresponding to each sign language action in the video. During the training phase, the model will be exposed to these annotated sign language videos and their corresponding sign language vocabulary sequences, and learn how to transform video frame sequences by optimizing the connection temporal classification (CTC) loss function. Mapping to sign language vocabulary sequences The CTC loss function allows the model to learn this mapping relationship without the need to accurately align each video frame, thereby achieving weakly supervised learning. The following formula is expressed as the calculation of the sign language video frame To sign language vocabulary sequence Probability , weakly supervised learning is achieved through the connectionist temporal classification (CTC) loss function:

[0033] in, represents a possible path from a sign language video frame to a sign language vocabulary, Represents the set of all possible paths.

[0034] Step S2 is a process of using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to the time step, wherein the process of using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to the time step includes: S21: performing encoding and decoding processing on the sign language video encoding features and the hidden layer vector related to the encoding and decoding attention mechanism through the encoding and decoding attention mechanism in the deep feedforward decoding module to generate a hidden state vector; S22: Performing a nonlinear transformation on the hidden state vector through a deep feedforward inference layer in the deep feedforward decoding module to generate a target hidden state vector; S23: performing a linear transformation through a linear layer in the deep feedforward decoding module to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; S24: normalizing the linearly transformed vector by a nonlinear function in the deep feedforward decoding module to generate a probability distribution of the mapped current vocabulary; S25: selecting a word with a probability greater than a preset probability from the current word as a predicted word according to the probability distribution; S26: Connecting the predicted words based on the time step and preset grammatical semantic rules to generate a predicted sentence.

[0035] Specifically, in this embodiment, the function of the module is to decode the video coding features obtained in the first step into the corresponding translation text. More specifically, step (1): the sign language video coding features obtained above are decoded into the corresponding translation text. Send to the attached Figure 3 The deep feedforward decoding module shown in Figure 3 As shown. Through the encoder-decoder multi-head attention mechanism in the deep feedforward decoding module For the hidden layer vector and the video feature vector generated by the encoder Processing to generate a new hidden state vector : ; The hidden layer vector It is gradually constructed and updated during the processing of the self-attention and encoder-decoder attention mechanisms of the decoder module, and it integrates word embedding information and encoded feature information.

[0036] Step (2): Through the deep feed-forward inference layer For the hidden state vector Perform nonlinear transformation to generate the final hidden state vector , referred to as the target hidden state vector in this embodiment; ; Step (3): Based on the target hidden state vector obtained in step (2) To predict words The probability of . First, the hidden state vector The linear transformation vector is formed by transforming through the linear layer, that is, mapping the features to the dimension of the vocabulary, and then the transformed vector is normalized by the SoftMax function to obtain the predicted probability distribution of each word at the current time step. The SoftMax function ensures that the sum of all probabilities is 1, so it can be interpreted as a probability distribution. According to this probability distribution, the word with the largest probability is selected as the predicted word at the current time step. In the decoding phase, the model generates words one by one and connects the words predicted at each time step in sequence to form a complete sentence. This process needs to consider the grammatical and semantic coherence between words, and the decoding process usually stops when a specific termination marker (EOS, indicating the end of the sentence) is encountered. At this time, the model has generated a complete sentence. In this specific implementation, the maximum likelihood estimation is used to obtain the sign language video frame. To the translated sentence The conditional probability , the calculation formula is as follows:

[0037] It should be noted that the deep feedforward encoding module and the deep feedforward decoding module are integrated in a pre-trained deep feedforward model; wherein, when pre-training the deep feedforward model, The target loss function of the deep feedforward model is the product of the first weight and the predicted sign language vocabulary loss plus the product of the second weight and the predicted sentence loss; wherein, The predicted sign language vocabulary loss is one minus the probability of the deep feedforward model correctly identifying the sign language vocabulary during training; The prediction sentence loss is one minus the probability that the deep feedforward model correctly confirms the predicted word from the target language vocabulary in the decoding time step during training.

[0038] Specifically, is the total objective loss function, which aims to adjust and The weight ratio enables the model to simultaneously translate sign language videos into sign language vocabulary and sentences. When the loss function no longer decreases significantly in multiple consecutive epochs, the training stops.

[0039]

[0040] in, is the weight value of the recognition loss function, which is the weight when testing sign language vocabulary, called the first weight. is the weight value of the translation loss function, that is, the weight when predicting a sentence, called the second weight.

[0041] It is the loss function of sign language recognition, that is, the loss function of the model to predict sign language vocabulary during training. This loss function is designed to measure the performance of the model in the sign language recognition task and minimize the recognition error.

[0042]

[0043] in, Indicates correct recognition of sign language vocabulary sequence probability.

[0044] It is the translation loss function, that is, the loss generated when generating a predicted sentence. This loss function aims to measure the performance of the model in the sign language translation task and minimize the translation error.

[0045]

[0046] in, represents the predicted word at decoding time step u The probability of Indicates the size of the target language vocabulary.

[0047] After repeated experiments, comparative experiments were conducted by setting the width of the deep feedforward inference layer to 512, 1024, 2048, 3072, 4096, and 5120 respectively. The experiment proved that when the width of the deep feedforward inference layer is 2048, the model performance is optimal. Comparative experiments were conducted by setting the number of layers of the deep feedforward inference model to 1, 2, 3, 4, 5, and 6 respectively. The experiment proved that when the number of layers of the deep feedforward inference model is 2, the model obtains better nonlinear expression ability and deep reasoning ability. Most existing sign language translation models adopt a single end-to-end translation architecture, which only supports the direct generation of natural language text from sign language videos. This embodiment can not only directly translate sign language videos into text end-to-end, but also translate them into sign language vocabulary and translated text. And two loss functions are used: recognition loss function (Formula as follows) and translation loss function (Formula as follows) , thereby conducting a comparative experiment through different weight ratios of sign language recognition loss function and sign language translation loss function, fixing the sign language translation loss value to 1, and the sign language recognition loss value to 0.5, 1, 2.5, 5, 10, and 20. The experiment proves that when the weight of sign language recognition loss and sign language translation loss is 10:1, the overall performance of the model is optimal. Therefore, in this embodiment, the deep feedforward inference layer of the deep feedforward model is 2048 layers; the number of layers of the deep feedforward inference model is two layers; the ratio of the first weight to the second weight is 10:1, so as to produce a more accurate prediction effect, which can greatly improve the translation indicators of BLEU, CHRF, and ROUGE, and improve the sign language prediction performance.

[0048] The meanings of the symbols in the above formula are as follows:

[0049]

[0050]

[0051] As described above, the method for translating sign language videos provided by the present invention can significantly enhance the model's ability to capture complex sign language features, especially the nonlinear relationship expression of sign language movements, through a deep feedforward encoding module and a deep feedforward decoding module, solve the problems of inaccurate semantic expression and missing detail information in existing models, and enable the generated text to better retain the semantic details in the sign language video and improve the credibility of the translation; the model not only supports the direct generation of natural language text, but also supports the generation of sign language vocabulary and natural language text; and optimizes the Transformer model structure for the sign language translation task, adjusts the width of the feedforward layer to 2048, the number of layers of the deep feedforward reasoning sign language translation model to 2 layers, and the weight of the sign language recognition loss and the sign language translation loss to 10:1, so that the optimized deep feedforward model is significantly superior to the prior art in translation indicators (such as BLEU, CHRF, ROUGE).

[0052] A sign language video translation system provided in an embodiment of the present application is introduced below. The sign language video translation system described below and the sign language video translation method described above can refer to each other.

[0053] like Figure 4 As shown, the present invention also provides a sign language video translation system 100, which can implement the sign language video translation method as described above, and the system includes a deep feedforward model; the deep feedforward model includes: A deep feedforward encoding module 101 is used to perform deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predict sign language vocabulary; A deep feedforward decoding module 102, configured to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connect the predicted words into predicted sentences according to time steps; The information integration module 103 is used to use the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

[0054] In this embodiment, the depth feedforward encoding module 101 includes: A frame division module 1011 is used to divide the acquired sign language video into frames to form continuous video frames; A feature extraction module 1012 is used to extract features from the video frame to obtain video features; A spatial encoding module 1013, configured to perform spatial embedding on the video features to form an embedding vector of a fixed dimension; A position coding module 1014, configured to add position coding to the embedding vector to generate an information video vector; A multi-head attention mechanism 1015, for performing multi-head feature attention calculation on the information video vector to obtain a preliminary feature vector; A deep feedforward reasoning layer 1016, for performing feature semantic capture on the preliminary feature vector to obtain sign language video coding features; The depth feedforward decoding module comprises: A codec attention mechanism 1021, used for performing codec processing on the sign language video coding features and the hidden layer vector related to the codec attention mechanism to generate a hidden state vector; A deep feedforward inference layer 1022, configured to perform a nonlinear transformation on the hidden state vector to generate a target hidden state vector; A linear layer 1023, configured to perform a linear transformation to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; A nonlinear function 1024, used to normalize the linearly transformed vector to generate a probability distribution of the mapped current vocabulary; A word prediction module 1025, configured to select a word with a high probability from the current word as a predicted word according to the probability distribution; The sentence prediction module 1026 is used to connect the predicted words based on the time step and the preset grammatical semantic rules to generate a predicted sentence.

[0055] The sign language video translation system 100 provided by the present invention generally performs calculations as in the above-mentioned sign language video translation method. For a more specific implementation process, reference may be made to the specific embodiments of the above-mentioned sign language video translation method.

[0056] As described above, the sign language video translation system 100 provided by the present invention, such as the sign language video translation method mentioned above, can also solve the problems of inaccurate semantic expression and missing detailed information in the existing model, and can enable the generated text to better retain the semantic details in the sign language video, improve the credibility of the translation, improve the translation indicators, and improve the translation effect.

[0057] The embodiment of the present invention further provides a computer-readable storage medium (not shown in the figure), which may be non-volatile or volatile, and stores a computer program, which, when executed by a processor, implements: Performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary; Using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps; The predicted sentence and the sign language vocabulary are used as a translation information file corresponding to the sign language video.

[0058] Specifically, the specific implementation method when the computer program is executed by the processor can refer to the description of the relevant steps in the translation method of the sign language video in the embodiment, which will not be repeated here.

[0059] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0060] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0062] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0063] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0064] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0065] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for methods, devices, electronic devices and computer-readable storage media, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The methods, devices, electronic devices and media described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0066] As described above with reference to the accompanying drawings, the translation method, system and storage medium of the sign language video proposed in the present invention are described by way of example. However, it should be understood by those skilled in the art that various improvements can be made to the translation method, system and storage medium of the sign language video proposed in the present invention without departing from the content of the present invention. Therefore, the protection scope of the present invention should be determined by the content of the attached claims.

Claims

1. A method for translating a sign language video, characterized in that: include: Performing deep feedforward encoding on the acquired sign language video through a preset deep feedforward encoding module to obtain sign language video encoding features and predict sign language vocabulary; Using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps; The predicted sentence and the sign language vocabulary are used as a translation information file corresponding to the sign language video.

2. The method for translating a sign language video according to claim 1, wherein: The method of performing deep feedforward encoding on the acquired sign language video by a preset deep feedforward encoding module to obtain the sign language video encoding features includes: Dividing the acquired sign language video into frames to form continuous video frames; Extracting features from the video frame using a preset feature extraction module to obtain video features; The video features are spatially embedded by a spatial encoding module in the deep feedforward encoding module to form an embedding vector of fixed dimension; adding a positional encoding to the embedding vector to generate an information video vector; Performing multi-head feature attention calculation on the information video vector through a preset multi-head attention mechanism to obtain a preliminary feature vector; The preliminary feature vector is semantically captured through a deep feedforward inference layer in the deep feedforward encoding module to obtain sign language video encoding features.

3. The method for translating a sign language video according to claim 2, wherein: The step of performing multi-head feature attention calculation on the information video vector by a preset multi-head attention mechanism to obtain a preliminary feature vector includes: Perform multi-head segmentation on the information video vector through the multi-head attention mechanism to obtain multiple subspaces; Performing self-attention calculation on the multiple subspaces to obtain similarity data of the information video vectors; The feature vectors of the multiple subspaces are fused based on the similarity data to form a preliminary feature vector.

4. The method for translating a sign language video according to claim 3, wherein: Obtaining the predicted sign language vocabulary includes: The predicted sign language vocabulary is matched according to the sign language video coding features through the sign language vocabulary architecture in the deep feedforward coding module; wherein, The sign language vocabulary framework is a pre-trained sign language vocabulary framework. When training the sign language vocabulary framework, the initial sign language vocabulary framework is repeatedly trained using pre-labeled sign language video samples and vocabulary sequences corresponding to the sign language video samples, and the vocabulary probability of the sign language video samples to the vocabulary sequence is calculated during each training process, and weakly supervised learning is performed on each round of training using the vocabulary probability.

5. The method for translating a sign language video according to claim 4, wherein: The method of using a preset deep feedforward decoding module to perform deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps, includes: Performing encoding and decoding processing on the sign language video encoding features and the hidden layer vector related to the encoding and decoding attention mechanism through the encoding and decoding attention mechanism in the deep feedforward decoding module to generate a hidden state vector; Performing a nonlinear transformation on the hidden state vector through a deep feedforward inference layer in the deep feedforward decoding module to generate a target hidden state vector; Performing a linear transformation through a linear layer in the deep feedforward decoding module to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; Normalizing the linearly transformed vector by a nonlinear function in the deep feedforward decoding module to generate a probability distribution of the mapped current vocabulary; Selecting a word with a probability greater than a preset probability from the current word as a predicted word according to the probability distribution; The predicted words are connected based on the time step and preset grammatical semantic rules to generate a predicted sentence.

6. The method for translating a sign language video according to claim 5, wherein: The deep feedforward encoding module and the deep feedforward decoding module are integrated in a pre-trained deep feedforward model; wherein, when pre-training the deep feedforward model, The target loss function of the deep feedforward model is the product of the first weight and the predicted sign language vocabulary loss plus the product of the second weight and the predicted sentence loss; wherein, The predicted sign language vocabulary loss is one minus the probability of the deep feedforward model correctly identifying the sign language vocabulary during training; The prediction sentence loss is one minus the probability that the deep feedforward model correctly confirms the predicted word from the target language vocabulary in the decoding time step during training.

7. The method for translating a sign language video according to claim 6, wherein: The deep feedforward inference layer of the deep feedforward model is 2048 layers; The number of layers of the deep feedforward model is two; The ratio of the first weight to the second weight is 10:

1.

8. A sign language video translation system, characterized in that: A method for translating a sign language video according to any one of claims 1 to 7 is implemented, wherein the system includes a deep feedforward model; the deep feedforward model includes: A deep feedforward encoding module, used for performing deep feedforward encoding on the acquired sign language video to obtain sign language video encoding features and predict sign language vocabulary; A deep feedforward decoding module, used for performing deep feedforward inference decoding on the sign language video coding features to obtain predicted words, and connecting the predicted words into predicted sentences according to time steps; The information integration module is used to use the predicted sentence and the sign language vocabulary as a translation information file corresponding to the sign language video.

9. The sign language video translation system according to claim 8, characterized in that: The deep feedforward encoding module comprises: A frame division module, used for dividing the acquired sign language video into frames to form continuous video frames; A feature extraction module, used for extracting features from the video frame to obtain video features; A spatial encoding module, used for spatially embedding the video features to form an embedding vector of fixed dimension; A position coding module, used for adding position coding to the embedding vector to generate an information video vector; A multi-head attention mechanism, for performing multi-head feature attention calculation on the information video vector to obtain a preliminary feature vector; A deep feed-forward inference layer, used for performing feature semantic capture on the preliminary feature vector to obtain sign language video coding features; The deep feedforward decoding module comprises: A codec attention mechanism is used to perform codec processing on the sign language video coding features and the hidden layer vector related to the codec attention mechanism to generate a hidden state vector; A deep feedforward inference layer, configured to perform a nonlinear transformation on the hidden state vector to generate a target hidden state vector; A linear layer, configured to perform a linear transformation to map the target hidden state vector to a preset vocabulary to form a linearly transformed vector; A nonlinear function, used for normalizing the linearly transformed vector to generate a probability distribution of the mapped current vocabulary; A word prediction module, used for selecting a word with a high probability from the current vocabulary as a predicted word according to the probability distribution; The sentence prediction module is used to connect the predicted words based on the time step and preset grammatical semantic rules to generate a predicted sentence.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program realizes the claims when executed by a processor A method for translating a sign language video as described in any one of the above.

Images