A semi-supervised sign language generation method, system and storage medium based on dual transformation

Abstract

The present invention discloses a semi-supervised sign language generation method, system and storage medium based on dual transformation, which belongs to the field of sign language generation and includes the following steps: 1. Use a complete training data set to perform supervised pre-training on a multi-task model G2P and a multi-modal model T2G; 2. Based on the duality of the tasks of the model G2P and the model T2G, use a training data set lacking labeled data to perform dual transformation training on the multi-task model G2P and the multi-modal model T2G, and design a multi-modal balance loss to balance the tendency of the prediction result to rely more on one modality data; 3. Perform knowledge distillation on the model T2G, and transfer the multi-modal knowledge in the model T2G to the student model t2g. The present invention proposes a dual transformation method for the semi-supervised sign language generation task for the first time, which achieves substantial progress compared to the most advanced methods currently available.

Description

Technical Field

[0001] The present invention relates to the field of sign language generation, and in particular to a semi-supervised sign language generation method, system and storage medium based on dual transformation. Background Art

[0002] The Sign Language Generation Task (SLP) aims to translate spoken text into corresponding continuous sign language sequences. The most popular method is to first translate the spoken text into sign language segments, and then generate sign language sequences from the sign language segments. This can be used to solve the problems of insufficient calibration of the two languages ​​and low learning efficiency in the process of directly translating spoken text into sign language sequences. However, the annotation of sign language segments in the training data set requires high professionalism in sign language and can only be obtained at a high cost. In contrast, there are a large number of spoken texts and sign language sequence videos on the Internet that are very easy to obtain, so the dual transformation method can be used to train the model. Summary of the invention

[0003] In response to the above problems, the present invention provides a semi-supervised sign language generation method, system and storage medium based on dual transformation to reduce the reliance on sign language segment annotation in sign language generation tasks. Compared with the current mainstream methods, the method proposed in the present invention only requires fewer sign language segment annotations.

[0004] To this end, the technical solution adopted in the present invention is as follows:

[0005] In a first aspect, the present invention provides a semi-supervised sign language generation method based on dual transformation, comprising the following steps:

[0006] S1: Use the first training data set to perform supervised pre-training on the multi-task model G2P. During the training process, the multi-task model G2P generates corresponding speech text prediction results and gesture sequence prediction results for the input sign language segment at the same time;

[0007] S2: using the speech text prediction results and gesture sequence prediction results obtained in step S1 as input samples and the real sign language segments in the first training data set as labels, the multimodal model T2G is supervised pre-trained;

[0008] S3: using the multimodal model T2G pre-trained in step S2 to predict the second training data set, and obtaining a prediction result of the sign language segment;

[0009] S4: using the second training data set and the sign language segment prediction result obtained in step S3, training the pre-trained multi-task model G2P;

[0010] S5: Use the multi-task model G2P trained in step S4 to predict the third training data set to obtain a speech text prediction result and a gesture sequence prediction result;

[0011] S6: training the pre-trained multimodal model T2G using the third training data set, the speech text prediction result obtained in step S5, and the gesture sequence prediction result;

[0012] S7: Use the first training data set to perform full-supervisory training again on the multi-task model G2P trained in step S4 and the multimodal model T2G trained in step S6;

[0013] S8: Use the multimodal model T2G trained in step S7 as the teacher model, perform knowledge distillation on the teacher model T2G, and transfer the knowledge to the student model t2g;

[0014] S9: Combine the multi-task model G2P trained in step S7 and the student model t2g obtained in step S8 to obtain the final sign language generation model, use the student model t2g in the final sign language generation model to translate the spoken text into sign language segments, and then use the multi-task model G2P to generate sign language sequences from the sign language segments.

[0015] In a second aspect, the present invention provides a semi-supervised sign language generation system based on dual transformation, which is used to implement the above-mentioned semi-supervised sign language generation method based on dual transformation.

[0016] In a third aspect, the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned semi-supervised sign language generation method based on dual transformation.

[0017] Compared with the prior art, the present invention proposes a novel dual transformation training method for balancing multimodal and multi-task. The dual training method makes full use of fewer sign language segment annotations to improve the performance of the two sub-task models. The proposed cross-modal balance loss further improves the overall performance of the system. Experimental data proves the excellence of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of the overall architecture of a semi-supervised sign language generation method based on dual transformation according to an exemplary embodiment;

[0019] Figure 2 is a schematic diagram of the architecture of a multimodal decoder according to an exemplary embodiment;

[0020] Figure 3 is a statistical graph showing the effect of the amount of training data annotation on model performance according to an exemplary embodiment;

[0021] Figure 4 The figure is a schematic diagram of a device terminal with data processing capability according to an exemplary embodiment. DETAILED DESCRIPTION

[0022] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The accompanying drawings are only schematic diagrams of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor systems and / or microcontroller systems.

[0023] The present invention represents the sign language generation task (SLP) as two stages, and divides the sign language generation model into a multimodal model T2G (Text to Gloss) and a multitask model G2P (Gloss to Pose); the present invention designs three training data sets, including the first training data set Second training dataset And the third training data set The first training data set consists of spoken text x, sign language fragment y, and gesture sequence z; the second training data set consists of text-gesture pairs and lacks sign language fragment annotations; the third training data set consists of only sign language fragment data.

[0024] The sign language generation model used in the present invention is denoted as DualSign. The goal of the present invention is to achieve a semi-supervised two-stage sign language generation task based on dual transformation, wherein T2G is used to translate spoken text into sign language segments, and G2P is used to generate sign language sequences from sign language segments.

[0025] Combination Figure 1 As shown, in a specific implementation of the present invention, a semi-supervised sign language generation method based on dual transformation includes the following steps:

[0026] S1: Using the first training dataset The multi-task model G2P is pre-trained under supervision. During the training process, the multi-task model G2P simultaneously generates corresponding speech text prediction results and gesture sequence prediction results for the input sign language fragments.

[0027] The multi-task model G2P consists of a sign language segment encoder θ enc , used to encode sign language fragments; a text decoder Used to generate spoken text based on the encoded sign language fragments; a gesture sequence decoder Used to generate a gesture sequence according to the encoded sign language segment; in this embodiment, the sign language segment encoder θ enc Contains several self-attention mechanism layers, cross-attention mechanism layers and feedforward neural networks; text decoder Contains a linear layer and a softmax layer; gesture sequence decoder It consists of a length normalization layer, several feed-forward transformer networks, and a gesture sequence generator.

[0028] The multi-task model G2P introduces an unsupervised continuous modeling mechanism to reduce the reliance on sign language segment annotations during training.

[0029] In a specific implementation of the present invention, step S1 includes the following specific contents:

[0030] The first training data set is represented as Where x1, y1, z1 represent the spoken text, sign language segment and gesture sequence in the first training data set respectively; They represent the spoken text, sign language segment and gesture sequence in the i-th training sample in the first training dataset respectively, N is the number of training samples in the first training dataset, W represents the number of spoken text words, V represents the number of sign language segments, L represents the number of gesture sequence frames, J represents the number of joint data in each frame, and D represents the dimension of the joint data.

[0031] During the training process, the sign language segment y1 is first used as input, and the multi-task model G2P is used to generate the spoken text prediction result And gesture sequence prediction results Then, the training loss is calculated based on the real speech text x1 and the real gesture sequence z1, and finally the parameters are updated to complete the training of the model G2P.

[0032] The training loss of the multi-task model is The calculation process is as follows:

[0033]

[0034]

[0035]

[0036]

[0037] in, Respectively represent the loss of generating spoken text and gesture sequence, represents the unsupervised continuous modeling mechanism loss during model training, λ d represents the weight factor, MSE is the mean square error loss, represents the total length of the target gesture sequence z1, d t Represents the length of the predicted t-th sign language segment.

[0038] S2: The predicted result of the spoken text obtained in step S1 And gesture sequence prediction results As input samples, the multimodal model T2G is used to predict the sign language segment and combined with the first training dataset The real sign language segment labels in complete the supervised pre-training of the multimodal model T2G.

[0039] In this step, the multimodal model T2G is used to input the spoken text and gesture sequences Generate the first prediction result of the sign language segment And the second prediction result of the sign language segment The multimodal model T2G comprises a gesture encoder, a text encoder, a cross-modal encoder and a multimodal decoder.

[0040] For the sake of convenience, the following takes the input speech text x and gesture sequence z as an example to introduce the calculation process of the multimodal model T2G, which mainly includes the following steps:

[0041] 2.1): Encode the input gesture sequence z through the gesture encoder to obtain gesture features Where d is the hidden dimension, T z is the total length of the gesture sequence z; in this embodiment, the gesture encoder includes a graph convolutional network and several feed-forward transformer networks;

[0042] 2.2): Encode the input speech text x through the text encoder to obtain text features Among them, T x is the number of words in the spoken text x; in this embodiment, the text encoder includes a text embedding layer and several feed-forward transformer networks;

[0043] 2.3): The feature h obtained in step 2.1) and step 2.2) is encoded by the cross-modal encoder pose and h text Perform multimodal feature fusion to finally obtain the fused gesture feature h′ containing high-level semantic information pose and fusion text feature h′ text ; In this embodiment, the cross-modal encoder includes several multimodal self-attention mechanism layers and a feedforward neural network. The specific calculation process is as follows:

[0044]

[0045] in, represents the concatenated gesture features and text features, ω q ,ω k ,ωv ∈R d×d is a learnable parameter, and the superscript T indicates transposition; the output of the last layer of the attention mechanism h′ is separated into fused gesture features h′ pose And fusion text features ht text .

[0046] 2.4): The fused gesture feature h′ obtained in step 2.3) is decoded by the multimodal decoder pose and fusion text feature h′ text Perform modal balancing processing; in this embodiment, the multimodal decoder includes a CTC joint training layer, a cross-attention decoupling layer, a gated bidirectional fusion layer, a sign language segment predictor, and several feedforward networks. The specific calculation process is as follows:

[0047] 2.4.1): Through the CTC joint training layer, the fusion gesture feature h′ pose Perform sign language segment prediction and obtain the first prediction result of the sign language segment

[0048] 2.4.2): Fusion gesture feature h′ through cross-attention decoupling layer pose and fusion text feature h′ text Perform decoupling operations based on the multi-head cross-attention mechanism to obtain the sign language segment-gesture cross-attention feature g′ <t,1 and the sign language segment-text cross-attention feature g′ ＜t,2 ,like Figure 2 As shown, the calculation process is as follows:

[0049] g′ ＜t,1 =MultiHead(g ＜t ,h′ pose ,h′ pose )

[0050] g′ ＜t,2 =MultiHead(g <t ,h′ text ,h′ text )

[0051] Among them, g ＜t ∈R t×d is the encoding feature of the first t real sign language segments, which is obtained by the sign language segment encoder θ in step S1 enc get.

[0052] 2.4.3): The sign language segment-gesture cross-attention feature g′ obtained in step 2.4.2) is fused through the gated bidirectional fusion layer ＜t,1 and the sign language segment-text cross-attention feature g′ <t,2Perform feature fusion to obtain the fused feature g′ <t , the calculation process is as follows:

[0053] g′ <t =g′ <t,1 ⊙c+g′ ＜t,2 ⊙(1-c)

[0054] c=σ([g′ <t,1 ; g′ <t,2 ; g ＜t ]W c )

[0055] Among them, ⊙ represents the Hadamard product operation, W c ∈R 3d×d is a learnable parameter, σ represents the sigmoid function, and [.;.;.] represents the concatenation operation.

[0056] 2.4.4): Fusion feature g′ through sign language segment predictor ＜t Make a prediction and get the second prediction result of the sign language segment

[0057] In this embodiment, the training loss during the modality balancing process of the multimodal decoder is The calculation process is as follows:

[0058]

[0059] Among them, n represents the number of multi-head cross-attention mechanism layers, g l ,g′ l represents the sign language segment-gesture cross-attention feature and sign language segment-text cross-attention feature of the lth layer, and cos represents the cosine similarity function.

[0060] 2.5) The second prediction result of the sign language segment obtained according to step 2.4) And the first prediction result of the sign language segment Combined with the true result y1, the training loss is calculated, and finally the parameters are updated to complete the training of the multimodal model T2G.

[0061] In this embodiment, the speech text prediction result obtained in step S1 is And gesture sequence prediction results As the input sample of the multimodal model T2G, the first prediction result of the sign language segment is generated through the above steps And the second prediction result of the sign language segment The training loss of the multimodal model T2G The calculation process is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] in, Sequence-to-sequence decoding loss, represents the decoding loss of the CTC joint training layer, represents the balance processing loss, λ b is the weight factor, φ(y1) is the set of CTC paths, and n represents the number of cross-attention mechanism layers. represents the sign language segment-gesture cross-attention feature and sign language segment-text cross-attention feature of the lth layer, and cos represents the cosine similarity function.

[0067] S3: Use the multimodal model T2G to train the second dataset Make predictions and get the prediction results of the sign language segment

[0068] In this step, the second training data set is represented as Contains M speech text and gesture sequence samples, and the prediction results of sign language segments

[0069] S4: Using the second training data set And the sign language segment prediction result obtained in step S3 Train the pre-trained multi-task model G2P.

[0070] In this embodiment, the training loss The calculation process is as follows:

[0071]

[0072] S5: Use the multi-task model G2P trained in step S4 to train the third training data set Make predictions and get the spoken text prediction results And gesture sequence prediction results

[0073] In this step, the third training data set Contains O sign language segment samples.

[0074] S6: Using the third training data set and the spoken text prediction result obtained in step S5 Gesture sequence prediction results Train the multimodal model T2G.

[0075] In this embodiment, the training loss The calculation process is as follows:

[0076]

[0077] S7: Using the first training data set The multi-task model G2P and the multimodal model T2G are trained again with full supervision.

[0078] In this embodiment, the total training loss The calculation process is as follows:

[0079]

[0080] Among them, α is the weight factor.

[0081] S8: Use the multimodal model T2G trained in step S7 as the teacher model, perform knowledge distillation on the teacher model T2G, and transfer the knowledge to the student model t2g.

[0082] In a specific implementation of the present invention, step S8 includes the following specific contents:

[0083] S8-1: Using the multimodal model T2G trained in step S7 to train the second training data set Generate K optimal sign language segment prediction results for each sample in Predict the sign language segment With the second sample Combine to form a new second training data set For the third training data set The same method is used for expansion, that is, the multi-task model G2P trained in step S7 is used to train the third training data set Generate K optimal spoken text prediction results for each sample in And gesture sequence prediction results Get a new third training data set Define the full dataset

[0084] S8-2: Perform knowledge distillation on the teacher model T2G on the full dataset S, knowledge distillation loss The calculation process is as follows:

[0085]

[0086] in, represents the t-th sign language segment y of the teacher model T2G t The output distribution of end-to-end decoding, P t2g (yt =k|y ＜t ,x;θ t2g ) is the conditional probability of the student model t2g, Represents the number of sign language segments in the full dataset S.

[0087] S8-3: The student model t2g obtained after knowledge distillation in step S8-2 is first trained on the first training data set Perform supervised training, training loss The calculation process is as follows:

[0088]

[0089] Combine the loss knowledge distillation loss in step S8-2 Get the total loss of the student model t2g The calculation process is as follows:

[0090]

[0091] Among them, λ is the weight factor.

[0092] S9: Combine the multi-task model G2P trained in step S7 and the student model t2g obtained in step S8 to obtain the final sign language generation model DualSign.

[0093] In the present embodiment, a semi-supervised sign language generation system based on dual transformation is also provided, and the system is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible.

[0094] The system comprises:

[0095] A first training module, which is used to perform supervised pre-training on the multi-task model G2P using the first training data set. During the training process, the multi-task model G2P simultaneously generates corresponding speech text prediction results and gesture sequence prediction results for the input sign language segment; and is used to use the obtained speech text prediction results and gesture sequence prediction results as input samples and the real sign language segments in the first training data set as labels to perform supervised pre-training on the multimodal model T2G;

[0096] A second training module, which is used to predict the second training data set using the pre-trained multimodal model T2G to obtain a sign language segment prediction result; and is used to train the pre-trained multi-task model G2P using the second training data set and the sign language segment prediction result obtained by the second training module;

[0097] A third training module, which is used to predict a third training data set using the multi-task model G2P trained by the second training module to obtain a speech text prediction result and a gesture sequence prediction result; and, used to train the multimodal model T2G trained by the first training module using the third training data set, the speech text prediction result and the gesture sequence prediction result obtained by the third training module;

[0098] A fourth training module, which is used to perform fully supervised training on the multi-task model G2P trained by the second training module and the multimodal model T2G trained by the third training module using the first training data set;

[0099] A knowledge distillation module is used to use the multimodal model T2G trained by the fourth training module as a teacher model, perform knowledge distillation on the teacher model T2G, and transfer the knowledge to the student model t2g;

[0100] The sign language generation model module is used to combine the multi-task model G2P trained by the fourth training module and the student model t2g obtained by the knowledge distillation module as the final sign language generation model; and use the student model t2g in the final sign language generation model to translate the spoken text into sign language fragments, and then use the multi-task model G2P to generate sign language sequences from the sign language fragments.

[0101] The implementation process of the functions and effects of each module in the above system is specifically described in the implementation process of the corresponding steps in the above method. For example, a specific process of the above system may be:

[0102] S1: Use the first training data set to perform supervised pre-training on the multi-task model G2P. During the training process, the multi-task model G2P generates corresponding speech text prediction results and gesture sequence prediction results for the input sign language segment at the same time;

[0103] S2: using the speech text prediction results and gesture sequence prediction results obtained in step S1 as input samples and the real sign language segments in the first training data set as labels, the multimodal model T2G is supervised pre-trained;

[0104] S3: using the multimodal model T2G pre-trained in step S2 to predict the second training data set, and obtaining a prediction result of the sign language segment;

[0105] S4: using the second training data set and the sign language segment prediction result obtained in step S3, training the pre-trained multi-task model G2P;

[0106] S5: Use the multi-task model G2P trained in step S4 to predict the third training data set to obtain a speech text prediction result and a gesture sequence prediction result;

[0107] S6: training the pre-trained multimodal model T2G using the third training data set, the speech text prediction result obtained in step S5, and the gesture sequence prediction result;

[0108] S7: Use the first training data set to perform full-supervisory training again on the multi-task model G2P trained in step S4 and the multimodal model T2G trained in step S6;

[0109] S8: Use the multimodal model T2G trained in step S7 as the teacher model, perform knowledge distillation on the teacher model T2G, and transfer the knowledge to the student model t2g;

[0110] S9: Combine the multi-task model G2P trained in step S7 and the student model t2g obtained in step S8 to obtain the final sign language generation model, use the student model t2g in the final sign language generation model to translate the spoken text into sign language segments, and then use the multi-task model G2P to generate sign language sequences from the sign language segments.

[0111] For the system embodiment, since it basically corresponds to the method embodiment, the specific implementation of each step can refer to the description of the method part, which will not be repeated here. The system embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0112] The embodiment of the semi-supervised sign language generation system based on dual transformation of the present invention can be applied to any device with data processing capability, and the device with data processing capability can be a device or apparatus such as a computer. The system embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capability in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From the hardware level, if Figure 4 As shown, it is a hardware structure diagram provided by this embodiment, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the system in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0113] An embodiment of the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned semi-supervised sign language generation method based on dual transformation is implemented.

[0114] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0115] The technical effect of the present invention is verified by experiments below.

[0116] In this embodiment, the PHOENIX14T dataset is used, which includes 8257 German texts, sign language segments and gesture sequences.

[0117] For the generated sign language segments, this embodiment uses BLEU-n and ROUGE evaluation indicators for evaluation; for the generated gesture sequences, this embodiment first back-translates the generated results to generate the corresponding speech text, and then uses BLEU-n, ROUGE, WER and DTW-MJE evaluation indicators for evaluation.

[0118] In this embodiment, the hidden dimension d is set to 256, λ d , b , λ, and α are 1.0, 1.0, 0.5, and 0.5 respectively, the learning rate is 0.001, and K is set to 5.

[0119] This embodiment proposes a model DualSign for semi-supervised sign language generation. In the specific experiment, 50% of the PHOENIX14T dataset is randomly selected as the training dataset. The remaining 50% is used as a sample and samples Semi-supervised training is performed; the fully supervised training directly uses the PHOENIX14T dataset as the training dataset. The standard transformer and NAT models are used as the baseline standard model (Baseline), and the dual training method (Baseline+DT) is added to the baseline as a comparison model. At the same time, it is compared with the most advanced models PT, MoMP and NAT-EA in recent years. The statistical results are shown in Tables 1, 2, and 3. B@n and R represent BLEU-n and ROUGE indicators, and the results in brackets are the experimental results produced by the teacher model T2G.

[0120] From Table 1, we can see the importance of the dual training method and multimodal balance processing method proposed in the present invention to the performance improvement of the model proposed in the present invention. The performance of the model DualSign proposed in the present invention even surpasses the Baseline based on fully supervised training in some aspects. This is due to the fact that the dual training method proposed in the present invention can use unlabeled data to improve the model performance.

[0121] As can be seen from Tables 2 and 3, the DualSign model proposed in this invention surpasses previous models with a significant advantage and reaches the most advanced performance level.

[0122] Table 1 Performance of DualSign, Baseline, and Baseline+DT on the PHOENIX14T dataset

[0123]

[0124] Table 2 Performance of each model in the task of converting sign language fragments to gesture sequences

[0125]

[0126] Table 3 Performance of each model in the task of speaking text to gesture sequence

[0127]

[0128] This example also explores the impact of the proportion of labeled data and the proportion of unlabeled data on model performance. The exploration results are as follows: Figure 3 As shown. Figure 3 (a) and (b) show that DualSign outperforms Baseline in both tasks at all scales, with the most notable difference occurring when the scale is about 30%. Figure 3(c) and (d) show that even without unlabeled data, DualSign can still outperform Baseline. With the increase of unlabeled data, both tasks can achieve better performance. This shows that the model proposed in this paper can also have good performance in the case of semi-supervised lack of labeled data.

[0129] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A semi-supervised sign language generation method based on dual transformation, characterized in that: The following steps are involved: S1: Use the first training data set to perform supervised pre-training on the multi-task model G2P. During the training process, the multi-task model G2P generates corresponding speech text prediction results and gesture sequence prediction results for the input sign language segment at the same time; The multi-task model G2P includes a sign language segment encoder, a text decoder and a gesture sequence decoder; The step S1 comprises: The first training data set is represented as Wherein, x1, y1, z1 represent the spoken text, sign language segment and gesture sequence in the first training data set respectively; denote the spoken text, sign language segment and gesture sequence in the i-th training sample in the first training data set, respectively, and N is the number of training samples in the first training data set; During the training process, an unsupervised continuous modeling mechanism is introduced to reduce the reliance on sign language segment annotations during training. First, the sign language segment y1 is used as input, and the speech text prediction result is generated through the multi-task model G2P. And gesture sequence prediction results Then, the training loss is calculated based on the real speech text x1 and the real gesture sequence z1, and finally the parameters are updated to complete the training of the model G2P; The training loss of the multi-task model is The calculation process is as follows: in, Respectively represent the loss of generating spoken text and gesture sequence, represents the unsupervised continuous modeling mechanism loss during model training, λ d represents the weight factor, MSE is the mean square error loss, represents the total length of the target gesture sequence z1, d t represents the length of the predicted t-th sign language segment, θ enc represents the parameters of the sign language segment encoder, Represents the text decoder parameters, represents the gesture sequence decoder parameters; S2: using the speech text prediction result and the gesture sequence prediction result obtained in step S1 as input samples and the real sign language segments in the first training data set as labels, the multimodal model T2G is supervised pre-trained; the multimodal model T2G comprises a gesture sequence encoder, a text encoder, a cross-modal encoder and a multimodal decoder; The step S2 comprises: 2.1): Encode the input gesture sequence through the gesture encoder to obtain the gesture feature h pose ; 2.2): Encode the input speech text through the text encoder to obtain the text feature h text ; 2.3): The feature h obtained in step 2.1) and step 2.2) is encoded by the cross-modal encoder pose and h te x t Perform multimodal feature fusion to finally obtain the fused gesture feature h′ containing high-level semantic information pose and fusion text feature h′ text ; The calculation process is as follows: Where h = [h pose ;h text ] represents the concatenated gesture features and text features, ω q ,ω k ,ω v is a learnable parameter, the superscript T indicates transposition, and h′ indicates the fused gesture feature and text feature; the final h′ is separated into the fused gesture feature h′ pose and fusion text feature h′ text ; 2.4): The fused gesture feature h′ obtained in step 2.3) is decoded by the multimodal decoder pose and fusion text feature h′ text Perform modal balancing to obtain the first prediction result of the sign language segment And the second prediction result of the sign language segment The multimodal decoder described in step 2.4) includes a CTC joint training layer, a cross-attention decoupling layer, a gated bidirectional fusion layer, and a sign language segment predictor. The specific calculation process is as follows: 2.4.1): Through the CTC joint training layer, the fusion gesture feature h′ pose Perform sign language segment prediction and obtain the first prediction result of the sign language segment 2.4.2): Fusion gesture feature h′ through cross-attention decoupling layer pose and fusion text feature h′ text Perform decoupling operations based on the multi-head cross-attention mechanism to obtain the sign language segment-gesture cross-attention feature g′ ＜t，1 and the sign language segment-text cross-attention feature g′ ＜t，2 , the calculation process is as follows: g′ ＜t，1 =MultiHead(g ＜t ,h′ pose ,h′ pose ) g′ ＜t，2 =MultiHead(g ＜t ,h′ text ,h′ text ) Among them, g ＜t ∈R t×d is the encoding feature of the first t real sign language segments; 2.4.3): The sign language segment-gesture cross-attention feature g′ obtained in step 2.4.2) is fused through the gated bidirectional fusion layer ＜t，1 and the sign language segment-text cross-attention feature g′ ＜t，2 Perform feature fusion to obtain the fused feature g′ ＜t , the calculation process is as follows: g′ ＜t =g′ ＜t，1 ⊙c+g′ ＜t，2 ⊙(1-c) c=σ([g′ ＜t，1 ;g′ ＜t，2 ;g ＜t ]W c ) Among them, ⊙ represents the Hadamard product operation, W c ∈R 3d×d is a learnable parameter, σ represents the sigmoid function, and [.;.;.] represents the concatenation operation; 2.4.4): Fusion feature g′ through sign language segment predictor ＜t Make a prediction and get the second prediction result of the sign language segment 2.5) The second prediction result of the sign language segment obtained according to step 2.4) And the first prediction result of the sign language segment Combined with the real result y1, the training loss is calculated, and finally the parameters are updated to complete the training of the multimodal model T2G; S3: using the multimodal model T2G pre-trained in step S2 to predict the second training data set, and obtaining a prediction result of the sign language segment; The second training data set described in step S3 is represented as The prediction result of the sign language segment is Wherein, x2 and z2 represent the spoken text and gesture sequence in the second training data set respectively; denote the spoken text and gesture sequence in the i-th training sample in the second training data set, respectively, M is the number of training samples in the second training data set, represents the prediction result of the sign language segment for the second training data set, represents the prediction result of the sign language segment for the i-th training sample in the second training data set; S4: using the second training data set and the sign language segment prediction result obtained in step S3, training the pre-trained multi-task model G2P; S5: Use the multi-task model G2P trained in step S4 to predict the third training data set to obtain a speech text prediction result and a gesture sequence prediction result; The third training data set described in step S5 is represented as Among them, y2 represents the sign language segment in the third training data set, represents the i-th sign language segment in the third training data set, and O is the number of training samples in the third training data set; S6: using the third training data set, the speech text prediction result obtained in step S5, and the gesture sequence prediction result, to train the pre-trained multimodal model T2G; S7: Use the first training data set to perform full-supervisory training again on the multi-task model G2P trained in step S4 and the multimodal model T2G trained in step S6; S8: Use the multimodal model T2G trained in step S7 as the teacher model, perform knowledge distillation on the teacher model T2G, and transfer the knowledge to the student model t2g; The step S8 comprises: S8-1: Generate K optimal sign language segment prediction results for each sample in the second training data set through the multimodal model T2G trained in step S7 Predict the sign language segment Combined with the second training data set to form a new second training data set The multi-task model G2P trained in step S7 generates K optimal speech text prediction results for each sample in the third training data set. And gesture sequence prediction results Get a new third training data set Define the full dataset Among them, x, y, and z represent the spoken text, sign language fragment, and gesture sequence in the full dataset respectively; S8-2: Use the multimodal model T2G trained in step S7 as the teacher model, perform knowledge distillation on the teacher model T2G on the full dataset S, and calculate the knowledge distillation loss S8-3: Perform supervised training on the first training data set for the student model t2g obtained after knowledge distillation in step S8-2, and calculate the student model training loss S8-4: Calculate the total loss: Among them, λ is the weight factor, Q s2s (y t =k|y <t , x, z; θ T2G ) represents the output distribution of the teacher model T2G for the t-th sign language segment yt in the full dataset through end-to-end decoding, P t2g (y t =k|y <t , x; θ t2g ) is the conditional probability of the student model t2g, Represents the number of sign language segments in the full dataset, represents the first training data set, x1, y1 represent the spoken text and sign language segments in the first training data set respectively, θ t2g represents the parameters of the student model t2g, θ T2G represents the parameters of the teacher model T2G; S9: Combine the multi-task model G2P trained in step S7 and the student model t2g obtained in step S8 to obtain the final sign language generation model, use the student model t2g in the final sign language generation model to translate the spoken text into sign language segments, and then use the multi-task model G2P to generate sign language sequences from the sign language segments.

2. A semi-supervised sign language generation system based on dual transformation, used to implement the semi-supervised sign language generation method according to claim 1, characterized in that: The system comprises: A first training module, which is used to perform supervised pre-training on the multi-task model G2P using the first training data set. During the training process, the multi-task model G2P simultaneously generates corresponding speech text prediction results and gesture sequence prediction results for the input sign language segment; and is used to use the obtained speech text prediction results and gesture sequence prediction results as input samples and the real sign language segments in the first training data set as labels to perform supervised pre-training on the multimodal model T2G; A second training module, which is used to predict the second training data set using the pre-trained multimodal model T2G to obtain a sign language segment prediction result; and is used to train the pre-trained multi-task model G2P using the second training data set and the sign language segment prediction result obtained by the second training module; A third training module, which is used to predict a third training data set using the multi-task model G2P trained by the second training module to obtain a speech text prediction result and a gesture sequence prediction result; and, used to train the multimodal model T2G trained by the first training module using the third training data set, the speech text prediction result and the gesture sequence prediction result obtained by the third training module; A fourth training module, which is used to perform fully supervised training on the multi-task model G2P trained by the second training module and the multimodal model T2G trained by the third training module using the first training data set; A knowledge distillation module is used to use the multimodal model T2G trained by the fourth training module as a teacher model, perform knowledge distillation on the teacher model T2G, and transfer the knowledge to the student model t2g; The sign language generation model module is used to combine the multi-task model G2P trained by the fourth training module and the student model t2g obtained by the knowledge distillation module as the final sign language generation model; and use the student model t2g in the final sign language generation model to translate the spoken text into sign language fragments, and then use the multi-task model G2P to generate sign language sequences from the sign language fragments.

3. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, it is used to implement the semi-supervised sign language generation method described in claim 1.

Images