Training method and device of end-to-end speech translation model and terminal equipment

Abstract

The invention provides a training method and device of an end-to-end speech translation model and terminal equipment, and relates to the technical field of speech translation. The method comprises the following steps: acquiring training data of end-to-end speech translation; initializing all parameters of the end-to-end speech translation model based on the pre-trained large model; according to the training data, training the initialized end-to-end speech translation model in a single-task training mode, a multi-task training mode and a self-learning training mode; and determining a trained end-to-end speech translation model according to the model parameters determined after training in each training mode. According to the scheme, all parameters of the end-to-end speech translation model are initialized through the pre-training large model, and new model parameters are not introduced in the training process, so that the training efficiency and the training effect of the model can be greatly improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of speech translation, and in particular to a training method, apparatus and terminal device for an end-to-end speech translation model. Background Art

[0002] Speech translation technology is the study of the technology of translating speech in one language into text in another language. As the use of end-to-end models in the field of natural language processing becomes more mature, end-to-end speech translation models are gradually being widely used by researchers.

[0003] However, due to the scarcity of end-to-end speech translation datasets, the quality of the trained end-to-end speech translation model needs to be improved. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a training method, apparatus and terminal device for an end-to-end speech translation model.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training an end-to-end speech translation model is provided, comprising:

[0006] Obtain training data for end-to-end speech translation;

[0007] Initialize all parameters of the end-to-end speech translation model based on the pre-trained large model;

[0008] According to the training data, the initialized end-to-end speech translation model is trained using single-task training, multi-task training, and self-learning training methods;

[0009] A trained end-to-end speech translation model is determined according to the model parameters determined after training in each training mode.

[0010] As an example, the end-to-end speech translation model includes a speech encoder, a text encoder, and a text decoder; the pre-trained large model includes a HuBERT model and an mBART model; all parameters of the end-to-end speech translation model are initialized based on the pre-trained large model, including:

[0011] Initialize all parameters of the speech encoder based on the HuBERT model;

[0012] All parameters of the text encoder and text decoder are initialized based on the mBART model.

[0013] In some embodiments of the present disclosure, the training data is triple data including source speech data, source text data and translated text data.

[0014] As a possible implementation method, the initialized end-to-end speech translation model is trained in a multi-task training manner according to the training data, including:

[0015] According to the source speech data and the source text data, the end-to-end speech translation model after the i-th round of training is trained for speech recognition task learning to obtain a first loss value; wherein i is an integer greater than or equal to 0, and when i=0, the end-to-end speech translation model after the i-th round of training is an initialized end-to-end speech translation model;

[0016] According to the source speech data and the translated text data, the end-to-end speech translation model after the i-th round of training is trained to perform speech translation task learning to obtain a second loss value;

[0017] According to the source text data and the translated text data, the end-to-end speech translation model after the i-th round of training is trained to perform machine translation task learning to obtain a third loss value;

[0018] Determining a multi-task loss value according to the first loss value, the second loss value, and the third loss value;

[0019] Based on the multi-task loss value, the model parameters of the end-to-end speech translation model after the i-th round of training are updated, the end-to-end speech translation model after the i+1-th round of training is determined, and the end-to-end speech translation model after the i+1-th round of training is continued to be multi-task trained until the end-to-end speech translation model after the multi-task training is obtained.

[0020] Wherein, determining the multi-task loss value according to the first loss value, the second loss value and the third loss value includes:

[0021] The first loss value, the second loss value, and the third loss value are weightedly summed to determine the multi-task loss value.

[0022] As an example, according to the source text data and the translated text data, the end-to-end speech translation model after the i-th round of training is subjected to machine translation task learning to obtain a third loss value, including:

[0023] Map the source text data to a high-dimensional feature space to obtain a source text feature vector;

[0024] Inputting the source text feature vector into the text encoder in the end-to-end speech translation model after the i-th round of training, and obtaining the machine translation data output by the text decoder in the end-to-end speech translation model after the i-th round of training;

[0025] A third loss value is obtained based on the translated text data and the machine translation data.

[0026] In some other embodiments of the present disclosure, the method further includes:

[0027] When performing speech translation task learning on the end-to-end speech translation model after the i-th round of training, obtaining a speech feature vector output by a speech encoder of the i-th end-to-end speech translation model;

[0028] Obtaining a fourth loss value according to the source text feature vector and the speech feature vector;

[0029] Wherein, determining the multi-task loss value according to the first loss value, the second loss value and the third loss value includes:

[0030] A multi-task loss value is determined according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0031] In some embodiments of the present disclosure, the training data further includes unsupervised source speech data; and according to the training data, the initialized end-to-end speech translation model is trained in a single-task and self-learning training mode, respectively, including:

[0032] The initialized end-to-end speech translation model is trained according to the source speech data and the translated text data to obtain an end-to-end speech translation model after single-task training;

[0033] Input the unsupervised source speech data into the end-to-end speech translation model trained with a single task to obtain the text data corresponding to the unsupervised speech data;

[0034] According to the source speech data, the translation text data, the unsupervised speech data and the corresponding text data, the end-to-end speech translation model after single-task training is trained to obtain the end-to-end speech translation model after self-learning training.

[0035] As a possible implementation, determining a trained end-to-end speech translation model according to model parameters determined after training in each training mode includes:

[0036] Determine target model parameters according to the model parameters determined after training in each training method;

[0037] The model parameters of the end-to-end speech translation model are updated to the target model parameters to obtain a trained end-to-end speech translation model.

[0038] Wherein, according to the model parameters determined after training in each training mode, the target model parameters are determined, including:

[0039] The model parameters determined after training in each training method are averaged to obtain the target model parameters.

[0040] In some embodiments of the present disclosure, non-streaming training is adopted when training the initialized end-to-end speech translation model, and the trained end-to-end speech translation model is an end-to-end speech translation model suitable for streaming decoding.

[0041] According to a second aspect of an embodiment of the present disclosure, there is provided an end-to-end speech translation method, comprising:

[0042] Obtaining voice data to be translated;

[0043] The speech data to be translated is input into a preset end-to-end speech translation model to obtain translated text data; wherein the end-to-end speech translation model is a model trained by the end-to-end speech translation model training method described in the first aspect above.

[0044] As a possible implementation manner, the step of inputting the speech data to be translated into a preset end-to-end speech translation model to obtain translated text data includes:

[0045] The speech data to be translated is input into the preset end-to-end speech translation model for streaming speech translation to obtain real-time translated text data.

[0046] According to a third aspect of an embodiment of the present disclosure, a training device for an end-to-end speech translation model is provided, comprising:

[0047] An acquisition module, used to acquire training data for end-to-end speech translation;

[0048] An initialization module, used to initialize all parameters of the end-to-end speech translation model based on the pre-trained large model;

[0049] A training module, used to train the initialized end-to-end speech translation model in single-task training, multi-task training and self-learning training modes according to the training data;

[0050] The determination module is used to determine the trained end-to-end speech translation model according to the model parameters determined after training in each training mode.

[0051] As an example, the end-to-end speech translation model includes a speech encoder, a text encoder, and a text decoder; the pre-trained large model includes a HuBERT model and an mBART model; and the initialization module is specifically used for:

[0052] Initialize all parameters of the speech encoder based on the HuBERT model;

[0053] All parameters of the text encoder and text decoder are initialized based on the mBART model.

[0054] In some embodiments of the present disclosure, the training data is triple data including source speech data, source text data and translated text data.

[0055] As a possible implementation, the training module includes a first training unit, and the first training unit is used to:

[0056] According to the source speech data and the source text data, the end-to-end speech translation model after the i-th round of training is trained for speech recognition task learning to obtain a first loss value; wherein i is an integer greater than or equal to 0, and when i=0, the end-to-end speech translation model after the i-th round of training is an initialized end-to-end speech translation model;

[0057] According to the source speech data and the translated text data, the end-to-end speech translation model after the i-th round of training is trained to perform speech translation task learning to obtain a second loss value;

[0058] According to the source text data and the translated text data, the end-to-end speech translation model after the i-th round of training is trained to perform machine translation task learning to obtain a third loss value;

[0059] Determining a multi-task loss value according to the first loss value, the second loss value, and the third loss value;

[0060] Based on the multi-task loss value, the model parameters of the end-to-end speech translation model after the i-th round of training are updated, the end-to-end speech translation model after the i+1-th round of training is determined, and the end-to-end speech translation model after the i+1-th round of training is continued to be multi-task trained until the end-to-end speech translation model after the multi-task training is obtained.

[0061] The first training unit is specifically used for:

[0062] The first loss value, the second loss value, and the third loss value are weightedly summed to determine the multi-task loss value.

[0063] In some embodiments of the present disclosure, the first training unit is specifically used for:

[0064] Map the source text data to a high-dimensional feature space to obtain a source text feature vector;

[0065] Inputting the source text feature vector into the text encoder in the end-to-end speech translation model after the i-th round of training, and obtaining the machine translation data output by the text decoder in the end-to-end speech translation model after the i-th round of training;

[0066] A third loss value is obtained based on the translated text data and the machine translation data.

[0067] In some other embodiments of the present disclosure, the first training unit is further used for:

[0068] Used to obtain a speech feature vector output by a speech encoder of the i-th end-to-end speech translation model when performing speech translation task learning on the end-to-end speech translation model after the i-th round of training;

[0069] Used to obtain a fourth loss value according to the source text feature vector and the speech feature vector;

[0070] A multi-task loss value is determined according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0071] As a possible implementation, the training data further includes unsupervised source speech data; the training module includes a second training unit, and the second training unit is specifically used for:

[0072] The initialized end-to-end speech translation model is trained according to the source speech data and the translated text data to obtain an end-to-end speech translation model after single-task training;

[0073] Input the unsupervised source speech data into the end-to-end speech translation model trained with a single task to obtain the text data corresponding to the unsupervised speech data;

[0074] According to the source speech data, the translation text data, the unsupervised speech data and the corresponding text data, the end-to-end speech translation model after single-task training is trained to obtain the end-to-end speech translation model after self-learning training.

[0075] In some embodiments of the present disclosure, the determination module is specifically used to:

[0076] Determine target model parameters according to the model parameters determined after training in each training method;

[0077] The model parameters of the end-to-end speech translation model are updated to the target model parameters to obtain a trained end-to-end speech translation model.

[0078] As an example, the determination module is also used to:

[0079] The model parameters determined after training in each training method are averaged to obtain the target model parameters.

[0080] In addition, non-streaming training is adopted when training the initialized end-to-end speech translation model, and the trained end-to-end speech translation model is an end-to-end speech translation model suitable for streaming decoding.

[0081] According to a fourth aspect of an embodiment of the present disclosure, there is provided an end-to-end speech translation device, comprising:

[0082] An acquisition module is used to acquire voice data with translation;

[0083] A translation module is used to input the speech data to be translated into a preset end-to-end speech translation model to obtain translated text data; wherein the end-to-end speech translation model is a model trained by the training method of the end-to-end speech translation model described in the first aspect above.

[0084] In some embodiments of the present disclosure, the translation module is specifically used to:

[0085] The speech data to be translated is input into a preset end-to-end speech translation model for streaming speech translation to obtain real-time translated text data.

[0086] According to a fifth aspect of an embodiment of the present disclosure, a terminal device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect above is implemented, and / or the method described in the second aspect above is implemented.

[0087] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the method described in the first aspect above, and / or implements the method described in the second aspect above.

[0088] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the method described in the first aspect above, and / or implements the method described in the second aspect above.

[0089] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: by initializing all parameters of the end-to-end speech translation model based on the pre-trained large model, and training the initialized end-to-end speech translation model in single-task training, multi-task training and self-learning training modes according to the training data, and determining the trained end-to-end speech translation model according to the model parameters determined after training in each training mode. This solution initializes all parameters of the end-to-end speech translation model by pre-training the large model, and the training process does not introduce new model parameters, which can greatly improve the training efficiency and training effect of the model. In addition, the model is trained in different training modes, and the trained end-to-end speech translation model is determined based on the model parameters determined by each training mode, which can further improve the training effect of the model, not only speeding up the training speed of the model, but also improving the translation quality and translation efficiency of the trained end-to-end speech translation model.

[0090] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0092] Figure 1 is a flow chart of a method for training an end-to-end speech translation model according to an exemplary embodiment;

[0093] Figure 2 is a flow chart of another method for training an end-to-end speech translation model according to an exemplary embodiment;

[0094] Figure 3 is a flowchart of another method for training an end-to-end speech translation model according to an exemplary embodiment;

[0095] Figure 4 is a flowchart of another method for training an end-to-end speech translation model according to an exemplary embodiment;

[0096] Figure 5 is a flow chart of an end-to-end speech translation method according to an exemplary embodiment;

[0097] Figure 6 is a structural block diagram of a training device for an end-to-end speech translation model according to an exemplary embodiment;

[0098] Figure 7 is a structural block diagram of an end-to-end speech translation device according to an exemplary embodiment;

[0099] Figure 8 It is a block diagram of a terminal device 800 according to an exemplary embodiment. DETAILED DESCRIPTION

[0100] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0101] It should be noted that speech translation technology is a technology that studies the translation of speech in one language into text in another language. As the use of end-to-end models in the field of natural language processing becomes more mature, end-to-end speech translation models are gradually being widely used by researchers. However, due to the scarcity of end-to-end speech translation datasets, the quality of trained end-to-end speech translation models needs to be improved.

[0102] In order to solve the above problems, the present disclosure provides a training method, apparatus and terminal device for an end-to-end speech translation model.

[0103] Figure 1 It is a flowchart of a method for training an end-to-end speech translation model according to an exemplary embodiment. It should be noted that the method for training an end-to-end speech translation model in the embodiment of the present disclosure can be applied to the training device for the end-to-end speech translation model in the embodiment of the present disclosure. The training device for the end-to-end speech translation model in the embodiment of the present disclosure can be configured in a terminal device, which can be a computing device such as a laptop computer, a desktop computer, a workbench, a server, etc. Figure 1 As shown, the training method of the end-to-end speech translation model includes the following steps:

[0104] Step 101: Acquire training data for end-to-end speech translation.

[0105] In some embodiments of the present disclosure, the training data for end-to-end speech translation may be triple data including source speech data, source text data, and translated text data, wherein the source text data refers to the text data corresponding to the source speech data, and the translated text data refers to the text data of the corresponding language translated from the source speech data. In embodiments of the present disclosure, the end-to-end speech translation model may be used for Chinese translation, English translation, etc. As an example, if the source speech data in the triple data is English speech data, the source text data is English text data corresponding to the English speech data, and the translated text data is Chinese translation text data corresponding to the English speech data.

[0106] As a possible implementation, the training data for end-to-end speech translation may be training data pre-set based on the training task. For example, the training data for end-to-end speech translation may be pre-stored in the terminal device, and when executing the training task, the terminal device obtains the training data from the corresponding path for training the end-to-end speech translation model.

[0107] Step 102: Initialize all parameters of the end-to-end speech translation model based on the pre-trained large model.

[0108] In some embodiments of the present disclosure, the pre-trained large model can be a model that has learned a large amount of language knowledge and language knowledge, and the structure of the pre-trained large model is consistent with the structure of the end-to-end speech translation model, and the parameters of the pre-trained large model can cover all parameters of the end-to-end speech translation model. It should be noted that the number of pre-trained large models can be one or more. Initializing all parameters of the end-to-end speech translation model based on the pre-trained large model is equivalent to applying the model parameters of the pre-trained large model to all parameters of the end-to-end speech translation model to obtain an initialized end-to-end speech translation model.

[0109] As a possible implementation, the end-to-end speech translation model may include a speech encoder, a text encoder and a text decoder, wherein the speech encoder may include a convolutional feature extraction layer and a Transformer encoder, and the text encoder and the text decoder may also be Transformer structures. The input data passes through the speech encoder and the text encoder to obtain a hidden state sequence, and then passes through the text decoder to obtain the corresponding decoding result. All parameters of the pre-trained large model are used to initialize the end-to-end speech translation model to give full play to the role and potential of the pre-trained model.

[0110] As an example, the pre-trained large model includes a HuBERT model and an mBART model, and the end-to-end speech translation model includes a speech encoder, a text encoder, and a text decoder. The implementation method of initializing all parameters of the end-to-end speech translation model based on the pre-trained large model includes: initializing all parameters of the speech encoder based on the HuBERT model; initializing all parameters of the text encoder and the text decoder based on the mBART model.

[0111] Step 103 , training the initialized end-to-end speech translation model in single-task training, multi-task training and self-learning training modes according to the training data.

[0112] That is to say, the initialized end-to-end speech translation model is trained separately in three different training methods using the training data.

[0113] Among them, single-task training refers to training the initialized end-to-end speech translation model only for speech translation, that is, performing speech translation training on the initialized end-to-end speech translation model based on the source speech data and the translated text data in the training data. For example, the source speech data in the training data is input into the initialized end-to-end speech translation model to obtain the speech translation text data output by the model, and the loss value is calculated based on the speech translation text data and the translated text data corresponding to the source speech data, and the model parameters are continuously updated based on the calculated loss value to obtain the end-to-end speech translation model after single-task training.

[0114] In some embodiments of the present disclosure, multi-task training refers to simultaneously performing multiple task learning on an initialized end-to-end speech translation model, so that the end-to-end speech translation model after multi-task training has the ability to handle multiple tasks. Among them, multi-task training can be training including speech translation tasks and speech recognition tasks, and multi-task training can also be training including speech translation tasks, speech recognition tasks and machine translation tasks. For example, the speech recognition task can be learned for the end-to-end speech translation model based on the source speech data and source text data in the training data, and the speech translation task can be learned for the end-to-end speech translation model based on the source speech data and translated text data in the training data. When calculating the loss value, the loss value of the multi-task training can be determined based on the loss value corresponding to the speech recognition task and the loss value corresponding to the speech translation task, and the model parameters are continuously updated based on the loss value of the multi-task training, and finally the end-to-end speech translation model after multi-task training is obtained.

[0115] In addition, the initialized end-to-end speech translation model is trained in a self-learning training mode, so that the trained model has self-learning ability to improve the translation ability of the end-to-end speech translation model. At the same time, due to the shortage of end-to-end speech translation training data, the model training effect can be improved by self-learning. As an example, the end-to-end speech translation model can be first trained with a single task of speech translation to obtain an end-to-end speech translation model after single-task training, and then the end-to-end speech translation model after single-task training is used to perform speech translation on the unsupervised source speech data to obtain the translation text data corresponding to the unsupervised speech data, and then the unsupervised source speech data and its corresponding translation text data are used as training sets to continue to train the end-to-end speech translation model after single-task training to obtain an end-to-end speech translation model after self-learning training.

[0116] As a possible implementation, if the end-to-end speech translation model includes a speech encoder, a text encoder and a text decoder, and the speech encoder includes a convolutional feature extraction layer and a transformer encoder. Among them, the speech encoder uses a two-layer CNN (convolutional neural network), the convolution kernel size is 2×2, and the dimension of attention in the network is set to 1024. The transformer encoder of the speech encoder is 24 layers, the text encoder is 12 layers, and the decoder has 12 layers. The calculation process of training the initialized end-to-end speech translation model may include: inputting the training data into the speech encoder, and obtaining a 1024-dimensional sequence as the output after the calculation of the convolution layer; adding absolute position coding after the output of the CNN; the self-attention sublayer stacked by the speech encoder calculates the sequence to obtain a high-dimensional hidden state sequence with acoustic information, and then enters the text encoder for semantic state calculation, and outputs a high-dimensional hidden state sequence; the decoding end adopts an autoregressive method, and each time receives the target word output last time and the entire high-dimensional hidden state sequence output by the encoding end as input, each sublayer of the decoding end mainly includes a Masked self-attention calculation unit and an attention calculation unit corresponding to the translation language, and the output of the last layer of the decoding end is passed to a matrix of 1024×the size of the vocabulary for calculation, and the calculation result is obtained by softmax (normalized exponential function) to obtain the probability of each position of the current decoded word in the dictionary; each decoding prediction outputs a target word, and the loss is calculated once for each batch, and the size of each batch is 40; through the Adam optimizer, the gradient is calculated by the gradient descent algorithm, and the model parameters are updated by back propagation.

[0117] Step 104: Determine a trained end-to-end speech translation model according to the model parameters determined after training in each training mode.

[0118] It can be understood that after training with the above three training methods, three end-to-end speech translation models are obtained, namely, the first end-to-end speech translation model, the second end-to-end speech translation model and the third end-to-end speech translation model, that is, three sets of model parameters can be determined. In order to enable the trained end-to-end speech translation model to have the capabilities of the first end-to-end speech translation model, the second speech translation model and the third speech translation model, the model parameters of the trained end-to-end speech translation model can be determined according to the model parameters of these three models to obtain the trained end-to-end speech translation model.

[0119] In some embodiments of the present disclosure, the implementation method of determining the trained end-to-end speech translation model according to the model parameters determined after training in each training method may include: determining the target model parameters according to the model parameters determined after training in each training method; updating the model parameters of the end-to-end speech translation model to the target model parameters to obtain the trained end-to-end speech translation model.

[0120] As an example, based on the model parameters determined after training in each training method, the implementation method of determining the target model parameters may include: weighted calculation of the model parameters of the first end-to-end speech translation model, the model parameters of the second end-to-end speech translation model, and the model parameters of the third end-to-end speech translation model, and determining the result of the weighted calculation as the target model parameters.

[0121] As another example, according to the model parameters determined after training in each training method, the target model parameters may be determined by averaging the model parameters determined after training in each training method to obtain the target model parameters.

[0122] As a possible implementation method, the training method of the end-to-end speech translation model of the embodiment of the present disclosure can use Pytorch to build a neural network framework based on the open source fairseq tool. Among them, the speech encoder of the end-to-end speech translation model uses a two-layer CNN, the convolution kernel size is 2×2, and the dimension of attention in the network is set to 1024. The transformer encoder of the speech encoder is 24 layers, the text encoder is 12 layers, and the number of layers of the decoder is 12 layers. During training, the original sampling points of the audio file corresponding to the source speech data are used as the input of the model, and the sampling rate is 16000Hz. In addition, during training, the subword level word segmentation method is used for decoding, and the mapping is based on the pre-trained mBART vocabulary. The original vocabulary includes 250*1000 words. In order to increase the training speed and reduce the model size, the training can be more stable. The vocabulary can be cropped based on the training data. For example, taking the Chinese and English training data as an example, the cropped vocabulary includes 64*1000 words, which can increase the training speed and reduce the model size.

[0123] In addition, during training, if the gradient of the convolutional feature extraction layer in the speech encoder is large, the gradient of this part can be scaled to make the training process more stable.

[0124] According to the training method of the end-to-end speech translation model of the embodiment of the present disclosure, all parameters of the end-to-end speech translation model are initialized based on the pre-trained large model, and the initialized end-to-end speech translation model is trained in single-task training, multi-task training and self-learning training modes according to the training data, and the trained end-to-end speech translation model is determined according to the model parameters determined after training in each training mode. This scheme initializes all parameters of the end-to-end speech translation model through the pre-trained large model to improve the training efficiency and training effect of the model. In addition, the model is trained in different training modes, and the trained end-to-end speech translation model is determined based on the model parameters determined by each training mode, which can further improve the training effect of the model, and can also speed up the training speed of the model, thereby improving the translation quality and translation efficiency of the trained end-to-end speech translation model.

[0125] Next, the process of multi-task training will be introduced in detail.

[0126] Figure 2 It is a flowchart of another training method of an end-to-end speech translation model according to an exemplary embodiment. It should be noted that the end-to-end speech translation model in the disclosed embodiment includes a speech encoder, a text encoder and a text decoder. All parameters of the speech encoder are initialized based on the HuBERT model, and all parameters of the text encoder and text decoder are initialized based on the mBART model. Figure 2 As shown, based on the above embodiment, according to the training data, the implementation method of training the initialized end-to-end speech translation model in a multi-task training manner may include the following steps:

[0127] Step 201, based on the source speech data and the source text data, the end-to-end speech translation model after the i-th round of training is subjected to speech recognition task learning to obtain a first loss value; wherein, i is an integer greater than or equal to 0, and when i=0, the end-to-end speech translation model after the i-th round of training is an initialized end-to-end speech translation model.

[0128] In some embodiments of the present disclosure, source speech data may be input into a speech encoder of an end-to-end speech translation model after the i-th round of training, and a speech recognition result may be obtained through a text encoder and a text decoder. Loss calculation may be performed on the speech recognition result and the corresponding source text data to obtain a first loss value.

[0129] Step 202: Perform speech translation task learning on the end-to-end speech translation model after the i-th round of training according to the source speech data and the translated text data to obtain a second loss value.

[0130] In some embodiments of the present disclosure, the source speech data can be input into the speech encoder of the end-to-end speech translation model after the i-th round of training, and the speech translation result can be obtained through the text encoder and the text decoder. The loss of the speech translation result and the corresponding translated text data is calculated to obtain a second loss value.

[0131] Step 203: Perform machine translation task learning on the end-to-end speech translation model after the i-th round of training according to the source text data and the translated text data to obtain a third loss value.

[0132] In some embodiments of the present disclosure, the implementation process of step 203 may include:

[0133] Step 203 - 1 , mapping the source text data to a high-dimensional feature space to obtain a source text feature vector.

[0134] Step 203-2: input the source text feature vector into the text encoder in the end-to-end speech translation model after the i-th round of training, and obtain the machine translation data output by the text decoder in the end-to-end speech translation model after the i-th round of training.

[0135] Step 203-3, obtaining a third loss value according to the translated text data and the machine translation data.

[0136] Step 204, determining a multi-task loss value according to the first loss value, the second loss value and the third loss value.

[0137] Step 205, based on the multi-task loss value, updates the model parameters of the end-to-end speech translation model after the i-th round of training, determines the end-to-end speech translation model after the i+1-th round of training, and continues to perform multi-task training on the end-to-end speech translation model after the i+1-th round of training until the end-to-end speech translation model after the multi-task training is obtained.

[0138] That is to say, the data in the triplet data is used to learn the corresponding tasks of the end-to-end speech translation model, and the multi-task loss value is calculated according to the loss value of each task, and the model is iteratively trained according to the multi-task loss value until the end-to-end speech translation model after multi-task training is obtained.

[0139] In some embodiments of the present disclosure, in the same round of training, the training data used in the training of the speech translation task, the speech recognition task, and the machine translation task are the same triplet of data. For example, if the training data is {S1, S2, ..., S n},S j is the jth set of triplet data in the training data, and S j =(X j ,Y j ,Z j), where X j is the source speech data in the jth group of triplet data, Y j is the translated text data in the jth group of triple data, Z j is the source text data in the jth group of triplet data. If S is used in the i+1th round of multi-task training of the end-to-end speech translation model j Triple data, according to X j and Z j Perform speech recognition task learning on the end-to-end speech translation model after the i-th round of training to obtain the first loss value. j and Y j Perform speech translation task learning on the end-to-end speech translation model after the i-th round of training to obtain the second loss value. j and Z j Perform machine translation task learning on the end-to-end speech translation model after the i-th round of training to obtain the third loss value.

[0140] The calculation of the first loss value, the second loss value and the third loss value can all be calculated by cross entropy. When performing multi-task training on the initialized end-to-end speech translation model, task differentiation can be performed by inputting tokens into the text decoder of the end-to-end speech translation model.

[0141] As a possible implementation, the implementation method of determining the multi-task loss value according to the first loss value, the second loss value and the third loss value in step 204 may be: weighted summing the first loss value, the second loss value and the third loss value to determine the multi-task loss value. For example, the first loss value, the second loss value and the third loss value may be summed, and the summed result may be determined as the multi-task loss value.

[0142] According to the training method of the end-to-end speech translation model of the embodiment of the present disclosure, when performing multi-task training on the initialized end-to-end speech translation model, the model is respectively trained on speech translation tasks, speech recognition tasks and machine translation tasks, so that the end-to-end speech translation model can learn knowledge related to the speech translation tasks, thereby improving the translation ability of the trained end-to-end speech translation model.

[0143] In order to further improve the training effect of the model, the present disclosure proposes another embodiment.

[0144] Figure 3 FIG. 1 is a flow chart of another method for training an end-to-end speech translation model according to an exemplary embodiment. Figure 3 As shown, based on the above embodiment, according to the training data, the implementation method of training the initialized end-to-end speech translation model in a multi-task training manner may include the following steps:

[0145] Step 301, based on the source speech data and the source text data, the end-to-end speech translation model after the i-th round of training is subjected to speech recognition task learning to obtain a first loss value; wherein, i is an integer greater than or equal to 0, and when i=0, the end-to-end speech translation model after the i-th round of training is an initialized end-to-end speech translation model.

[0146] Step 302: Perform speech translation task learning on the end-to-end speech translation model after the i-th round of training according to the source speech data and the translated text data to obtain a second loss value.

[0147] Step 303: Perform machine translation task learning on the end-to-end speech translation model after the i-th round of training according to the source text data and the translated text data to obtain a third loss value.

[0148] The implementation process of step 303 may include:

[0149] Step 303 - 1 , mapping the source text data to a high-dimensional feature space to obtain a source text feature vector.

[0150] Step 303 - 2 , input the source text feature vector into the text encoder in the end-to-end speech translation model after the i-th round of training, and obtain the machine translation data output by the text decoder in the end-to-end speech translation model after the i-th round of training.

[0151] Step 303-3, obtaining a third loss value according to the translated text data and the machine translation data.

[0152] Step 304, when performing speech translation task learning on the end-to-end speech translation model after the i-th round of training, obtaining a speech feature vector output by a speech encoder of the end-to-end speech translation model after the i-th round of training.

[0153] Step 305: Obtain a fourth loss value according to the source text feature vector and the speech feature vector.

[0154] That is to say, the loss is calculated based on the text features of the translated text data and the speech features obtained by the model when performing speech translation on the source speech data, so as to achieve the purpose of comparative learning, thereby making the language features output by the speech encoder more accurate and improving the training effect of the model.

[0155] Step 306, determining a multi-task loss value according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0156] In some embodiments of the present disclosure, the first loss value, the second loss value, the third loss value, and the fourth loss value, and the implementation method of determining the multi-task loss value may include: weighted summing the first loss value, the second loss value, the third loss value, and the fourth loss value, and determining the weighted summation result as the multi-task loss value. As an example, the first loss value, the second loss value, the third loss value, and the fourth loss value may be summed, and the summation result may be used as the multi-task loss value.

[0157] Step 307, based on the multi-task loss value, the model parameters of the end-to-end speech translation model after the i-th round of training are updated, the end-to-end speech translation model after the i+1-th round of training is determined, and the end-to-end speech translation model after the i+1-th round of training is continued to be multi-task trained until the end-to-end speech translation model after the multi-task training is obtained.

[0158] According to the training method of the end-to-end speech translation model of the embodiment of the present disclosure, contrastive learning is introduced when performing multi-task training on the initialized end-to-end speech translation model. The fourth loss value is calculated by contrastive learning, and the multi-task loss value is determined by combining the first loss value, the second loss value, the third loss value and the fourth loss value to improve the effect of the end-to-end speech translation model after the multi-task training, thereby improving the quality of the end-to-end speech translation model after the final training.

[0159] Next, the implementation process of single-task training and self-learning training will be introduced in detail.

[0160] Figure 4 FIG. 1 is a flowchart of another method for training an end-to-end speech translation model according to an exemplary embodiment. In the disclosed embodiment, the training data also includes unsupervised source speech data. Figure 4 As shown, based on the above embodiment, according to the training data, the initialized end-to-end speech translation model is trained in a single-task and self-learning training mode, including:

[0161] Step 401: Train the initialized end-to-end speech translation model according to the source speech data and the translation text data to obtain the end-to-end speech translation model after single-task training.

[0162] Step 402: input the unsupervised source speech data into the end-to-end speech translation model after single-task training to obtain text data corresponding to the unsupervised speech data.

[0163] Step 403, training the end-to-end speech translation model after single-task training according to the source speech data, the translation text data, the unsupervised speech data and the corresponding text data, to obtain the end-to-end speech translation model after self-learning training.

[0164] It can be understood that the end-to-end speech translation model after single-task training already has a certain speech translation capability. Therefore, the end-to-end speech translation model after single-task training is used to perform speech translation on the unsupervised source speech data to obtain the text data corresponding to the unsupervised source speech data. Then, the supervised data and unsupervised speech data in the training data and their corresponding text data are used to continue training the end-to-end speech translation model after single-task training. This allows the model to have the ability to translate speech while also having the ability to self-learn, thereby improving the training effect of the model and also improving the translation quality and efficiency of the trained model.

[0165] In combination with the above embodiments, Figure 1 The implementation process of step 104 may include: determining the model parameters of the end-to-end speech translation model after single-task training as first model parameters; determining the model parameters of the end-to-end speech translation model after multi-task training as second model parameters; determining the model parameters of the end-to-end speech translation model after self-learning training as third model parameters; averaging the first model parameters, the second model parameters and the third model parameters to obtain target model parameters; updating the model parameters of the end-to-end speech translation model to the target model parameters to obtain the trained end-to-end speech translation model.

[0166] According to the training method of the end-to-end speech translation model of the embodiment of the present disclosure, the unsupervised source speech data is speech translated by the end-to-end speech translation model after single-task training to obtain text data corresponding to the unsupervised speech data, and then the end-to-end speech translation model after single-task training is further trained based on the supervised data and the unsupervised source speech data and their corresponding text data, so that the trained model has the ability of speech translation and the ability of self-learning, so as to improve the training effect of the model.

[0167] It should be noted that non-streaming training is used when training the initialized end-to-end speech translation model, and the trained end-to-end speech translation model is an end-to-end speech translation model suitable for streaming decoding. Non-streaming training means that translation starts after the source speech data is input during the training process, rather than translating while inputting, that is, translation is performed after a complete sentence is finished. Streaming decoding means that when decoding is performed based on the trained end-to-end speech translation model, the wait-k strategy can be used, which is usually implemented based on a decoding tool, such as decoding through the SimulEval decoding tool. The use of non-streaming training and streaming decoding can greatly simplify the training process without compromising the quality of translation.

[0168] Next, the present disclosure also proposes an end-to-end speech translation method.

[0169] Figure 5FIG. 1 is a flow chart of an end-to-end speech translation method according to an exemplary embodiment. Figure 5 As shown, the end-to-end speech translation method may include the following steps:

[0170] Step 501: Acquire speech data to be translated.

[0171] In some embodiments of the present disclosure, the voice data to be translated may be real-time voice data in an ongoing meeting, voice data in a video, or other forms of voice data. The voice data to be translated may be obtained by obtaining real-time voice data to be translated based on a microphone, or by reading a file containing voice data, or by other means, which is not limited by the present disclosure.

[0172] Step 502: input the speech data to be translated into a preset end-to-end speech translation model to obtain translated text data.

[0173] The end-to-end speech translation model is a model trained by the end-to-end speech translation model training method of the above embodiment.

[0174] In some embodiments of the present disclosure, the voice data to be translated may also be input into a preset end-to-end voice translation model for streaming voice translation to obtain real-time translated text data.

[0175] According to the end-to-end speech translation method of the embodiment of the present disclosure, the speech data with translation is translated by the trained end-to-end speech translation model to obtain the translated text data. Since the trained end-to-end speech translation model has high translation quality and fast translation speed, the present solution can improve the quality and efficiency of speech translation by the trained end-to-end speech translation model.

[0176] In order to implement the above embodiments, the present disclosure proposes a training device for an end-to-end speech translation model.

[0177] Figure 6 FIG. 1 is a structural block diagram of a training device for an end-to-end speech translation model according to an exemplary embodiment. Figure 6 As shown, the device includes: an acquisition module 610, an initialization module 620, a training module 630, and a determination module 640.

[0178] in:

[0179] An acquisition module 610 is used to acquire training data for end-to-end speech translation;

[0180] An initialization module 620, used to initialize all parameters of the end-to-end speech translation model based on the pre-trained large model;

[0181] A training module 630 is used to train the initialized end-to-end speech translation model in single-task training, multi-task training and self-learning training modes according to the training data;

[0182] The determination module 640 is used to determine the trained end-to-end speech translation model according to the model parameters determined after training in each training mode.

[0183] As an example, the end-to-end speech translation model includes a speech encoder, a text encoder, and a text decoder; the pre-trained large model includes a HuBERT model and an mBART model; the initialization module 620 is specifically used for:

[0184] Initialize all parameters of the speech encoder based on the HuBERT model;

[0185] All parameters of the text encoder and text decoder are initialized based on the mBART model.

[0186] In some embodiments of the present disclosure, the training data is triple data including source speech data, source text data and translated text data.

[0187] As a possible implementation, the training module 630 includes a first training unit 631, and the first training unit 631 is used to:

[0188] According to the source speech data and the source text data, the end-to-end speech translation model after the i-th round of training is trained for speech recognition task learning to obtain a first loss value; wherein i is a positive integer, and when i=1, the end-to-end speech translation model after the i-th round of training is an initialized end-to-end speech translation model;

[0189] According to the source speech data and the translated text data, the end-to-end speech translation model after the i-th round of training is trained to perform speech translation task learning to obtain a second loss value;

[0190] According to the source text data and the translated text data, the end-to-end speech translation model after the i-th round of training is trained to perform machine translation task learning to obtain a third loss value;

[0191] Determining a multi-task loss value according to the first loss value, the second loss value, and the third loss value;

[0192] Based on the multi-task loss value, the model parameters of the end-to-end speech translation model after the i-th round of training are updated, the end-to-end speech translation model after the i+1-th round of training is determined, and the end-to-end speech translation model after the i+1-th round of training is continued to be multi-task trained until the end-to-end speech translation model after the multi-task training is obtained.

[0193] As an example, the first training unit 631 is specifically used for:

[0194] The first loss value, the second loss value, and the third loss value are weightedly summed to determine the multi-task loss value.

[0195] In some embodiments of the present disclosure, the first training unit 631 is specifically used to:

[0196] Map the source text data to a high-dimensional feature space to obtain a source text feature vector;

[0197] Inputting the source text feature vector into the text encoder in the end-to-end speech translation model after the i-th round of training, and obtaining the machine translation data output by the text decoder in the end-to-end speech translation model after the i-th round of training;

[0198] A third loss value is obtained based on the translated text data and the machine translation data.

[0199] In some other embodiments of the present disclosure, the first training unit 631 is further used for:

[0200] When performing speech translation task learning on the end-to-end speech translation model after the i-th round of training, obtaining a speech feature vector output by a speech encoder of the end-to-end speech translation model after the i-th round of training;

[0201] Obtaining a fourth loss value according to the source text feature vector and the speech feature vector;

[0202] A multi-task loss value is determined according to the first loss value, the second loss value, the third loss value, and the fourth loss value.

[0203] As a possible implementation, the training data further includes unsupervised source speech data; the training module 630 includes a second training unit 632, and the second training unit 632 is specifically used for:

[0204] The initialized end-to-end speech translation model is trained according to the source speech data and the translated text data to obtain an end-to-end speech translation model after single-task training;

[0205] Input the unsupervised source speech data into the end-to-end speech translation model trained with a single task to obtain the text data corresponding to the unsupervised speech data;

[0206] According to the source speech data, the translation text data, the unsupervised speech data and the corresponding text data, the end-to-end speech translation model after single-task training is trained to obtain the end-to-end speech translation model after self-learning training.

[0207] In some embodiments of the present disclosure, the determination module 640 is specifically used to:

[0208] Determine target model parameters according to the model parameters determined after training in each training method;

[0209] The model parameters of the end-to-end speech translation model are updated to the target model parameters to obtain a trained end-to-end speech translation model.

[0210] As an example, the determination module 640 is further configured to:

[0211] The model parameters determined after training in each training method are averaged to obtain the target model parameters.

[0212] In addition, non-streaming training is adopted when training the initialized end-to-end speech translation model, and the trained end-to-end speech translation model is an end-to-end speech translation model suitable for streaming decoding.

[0213] According to the training device of the end-to-end speech translation model of the embodiment of the present disclosure, all parameters of the end-to-end speech translation model are initialized based on the pre-trained large model, and the initialized end-to-end speech translation model is trained in single-task training, multi-task training and self-learning training modes according to the training data, and the trained end-to-end speech translation model is determined according to the model parameters determined after training in each training mode. This scheme initializes all parameters of the end-to-end speech translation model through the pre-trained large model to improve the training efficiency and training effect of the model. In addition, the training effect of the model can be further improved by training the model in different training modes, and the training speed of the model can be accelerated, thereby improving the translation quality and translation efficiency of the trained model.

[0214] In order to implement the above embodiments, the present disclosure provides an end-to-end speech translation device.

[0215] Figure 7 FIG. 1 is a structural block diagram of an end-to-end speech translation device according to an exemplary embodiment. Figure 7 As shown, the device includes an acquisition module 710 and a translation module 720. Among them,

[0216] An acquisition module 710 is used to acquire voice data with translation;

[0217] The translation module 720 is used to input the speech data to be translated into a preset end-to-end speech translation model to obtain translated text data; wherein the end-to-end speech translation model is a model trained by the training method of the end-to-end speech translation model in the above embodiment.

[0218] In some embodiments of the present disclosure, the translation module 720 is specifically used to:

[0219] The speech data to be translated is input into the preset end-to-end speech translation model for streaming speech translation to obtain real-time translated text data.

[0220] According to the end-to-end speech translation device of the embodiment of the present disclosure, the speech data with translation is translated through the trained end-to-end speech translation model to obtain the translated text data. Since the trained end-to-end speech translation model has high translation quality and fast translation speed, the present solution can improve the quality and efficiency of speech translation through the trained end-to-end speech translation model.

[0221] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0222] In order to implement the above embodiments, the present disclosure provides a terminal device, a computer-readable storage medium, and a computer program product.

[0223] Figure 8 800 is a block diagram of a terminal device 800 according to an exemplary embodiment. It should be noted that the terminal device 800 can be used to implement the training method of the end-to-end speech translation model of the embodiment of the present disclosure, and / or implement the end-to-end speech translation method of the embodiment of the present disclosure. For example, the terminal device 800 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0224] Reference Figure 8 The terminal device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0225] The processing component 802 generally controls the overall operation of the terminal device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0226] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the terminal device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0227] The power component 806 provides power to various components of the terminal device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device 800.

[0228] The multimedia component 808 includes a screen that provides an output interface between the terminal device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0229] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the terminal device 800 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0230] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0231] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the terminal device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of the components, such as the display and keypad of the terminal device 800, and the sensor assembly 814 can also detect the position change of the terminal device 800 or a component of the terminal device 800, the presence or absence of contact between the user and the terminal device 800, the orientation or acceleration / deceleration of the terminal device 800, and the temperature change of the terminal device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0232] The communication component 816 is configured to facilitate the communication between the terminal device 800 and other devices in a wired or wireless manner. The terminal device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0233] In an exemplary embodiment, the terminal device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0234] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of a terminal device 800 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0235] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0236] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A training method for an end-to-end speech translation model, characterized in that: include: Obtain training data for end-to-end speech translation; Initialize all parameters of the end-to-end speech translation model based on the pre-trained large model; According to the training data, the initialized end-to-end speech translation model is trained by single-task training, multi-task training and self-learning training respectively; According to the model parameters determined after training in each of the training modes, a trained end-to-end speech translation model is determined.

2. The method according to claim 1, characterized in that The end-to-end speech translation model includes a speech encoder, a text encoder and a text decoder; the pre-trained large model includes a HuBERT model and an mBART model; and initializing all parameters of the end-to-end speech translation model based on the pre-trained large model includes: Initializing all parameters of the speech encoder based on the HuBERT model; All parameters of the text encoder and text decoder are initialized based on the mBART model.

3. The method according to claim 2, characterized in that The training data is triple data including source speech data, source text data and translated text data.

4. The method according to claim 3, characterized in that According to the training data, the initialized end-to-end speech translation model is trained in a multi-task training manner, including: According to the source speech data and the source text data, the end-to-end speech translation model after the i-th round of training is subjected to speech recognition task learning to obtain a first loss value; wherein i is an integer greater than or equal to 0, and when i=0, the end-to-end speech translation model after the i-th round of training is the initialized end-to-end speech translation model; According to the source speech data and the translated text data, performing speech translation task learning on the end-to-end speech translation model after the i-th round of training to obtain a second loss value; According to the source text data and the translated text data, performing machine translation task learning on the end-to-end speech translation model after the i-th round of training to obtain a third loss value; Determining a multi-task loss value according to the first loss value, the second loss value, and the third loss value; Based on the multi-task loss value, the model parameters of the end-to-end speech translation model after the i-th round of training are updated, the end-to-end speech translation model after the i+1-th round of training is determined, and multi-task training is continued for the end-to-end speech translation model after the i+1-th round of training until the end-to-end speech translation model after the multi-task training is obtained.

5. The method according to claim 4, characterized in that The determining of the multi-task loss value according to the first loss value, the second loss value and the third loss value comprises: The first loss value, the second loss value and the third loss value are weightedly summed to determine the multi-task loss value.

6. The method according to claim 4, characterized in that The step of performing machine translation task learning on the end-to-end speech translation model after the i-th round of training according to the source text data and the translated text data to obtain a third loss value includes: Mapping the source text data to a high-dimensional feature space to obtain a source text feature vector; Inputting the source text feature vector into the text encoder in the end-to-end speech translation model after the i-th round of training, and obtaining machine translation data output by the text decoder in the end-to-end speech translation model after the i-th round of training; The third loss value is obtained according to the translated text data and the machine translation data.

7. The method according to claim 6, characterized in that The method further comprises: When performing speech translation task learning on the end-to-end speech translation model after the i-th round of training, obtaining a speech feature vector output by a speech encoder of the end-to-end speech translation model after the i-th round of training; Acquire a fourth loss value according to the source text feature vector and the speech feature vector; Wherein, determining the multi-task loss value according to the first loss value, the second loss value and the third loss value includes: The multi-task loss value is determined according to the first loss value, the second loss value, the third loss value and the fourth loss value.

8. The method according to claim 3, characterized in that The training data also includes unsupervised source speech data; according to the training data, the initialized end-to-end speech translation model is trained in a single-task and self-learning training mode, including: Training the initialized end-to-end speech translation model according to the source speech data and the translated text data to obtain an end-to-end speech translation model after single-task training; Inputting the unsupervised source speech data into the end-to-end speech translation model after the single-task training to obtain text data corresponding to the unsupervised speech data; The end-to-end speech translation model after the single-task training is trained according to the source speech data, the translated text data, the unsupervised speech data and the text data corresponding thereto, so as to obtain the end-to-end speech translation model after the self-learning training.

9. The method according to claim 1, characterized in that: Determining the trained end-to-end speech translation model according to the model parameters determined after training in each of the training modes includes: Determine target model parameters according to the model parameters determined after training in each of the training methods; The model parameters of the end-to-end speech translation model are updated to the target model parameters to obtain the trained end-to-end speech translation model.

10. The method according to claim 9, characterized in that Determining target model parameters according to the model parameters determined after training in each of the training modes includes: The model parameters determined after training in each of the training modes are averaged to obtain the target model parameters.

11. The method according to any one of claims 1 to 10, characterized in that Non-streaming training is adopted when training the initialized end-to-end speech translation model, and the trained end-to-end speech translation model is an end-to-end speech translation model suitable for streaming decoding.

12. An end-to-end speech translation method, characterized in that: include: Obtaining voice data to be translated; The speech data to be translated is input into a preset end-to-end speech translation model to obtain translated text data; wherein the end-to-end speech translation model is a model trained by the training method of the end-to-end speech translation model described in any one of claims 1-11.

13. The method according to claim 12, characterized in that The step of inputting the speech data to be translated into a preset end-to-end speech translation model to obtain translated text data includes: The speech data to be translated is input into a preset end-to-end speech translation model for streaming speech translation to obtain real-time translated text data.

14. A training device for an end-to-end speech translation model, characterized in that: include: An acquisition module, used to acquire training data for end-to-end speech translation; An initialization module, used to initialize all parameters of the end-to-end speech translation model based on the pre-trained large model; A training module, used to train the initialized end-to-end speech translation model in single-task training, multi-task training and self-learning training modes according to the training data; The determination module is used to determine the trained end-to-end speech translation model according to the model parameters determined after training in each of the training methods.

15. An end-to-end speech translation device, characterized in that: include: An acquisition module is used to acquire voice data with translation; A translation module is used to input the speech data to be translated into a preset end-to-end speech translation model to obtain translated text data; wherein the end-to-end speech translation model is a model trained by the training method of the end-to-end speech translation model described in any one of claims 1-11.

16. A terminal device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented, and / or the method according to any one of claims 12 to 13 is implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented, and / or the method according to any one of claims 12 to 13 is implemented.

18. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 11 and / or the method according to any one of claims 12 to 13.

Images