Training method, speech translation method, device and computer readable medium

Abstract

The present disclosure provides a training method for a speech translation model, a speech translation method, a device and a computer-readable medium. The training method includes: obtaining a source speech representation sequence corresponding to source speech data and a source text representation sequence corresponding to the source speech data; obtaining a mixed sequence according to the source speech representation sequence and the source text representation sequence; processing the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and processing the mixed sequence using the speech translation model to output a second probability distribution of a second target text; calculating a total loss function according to the first probability distribution and the second probability distribution; and training the speech translation model according to the total loss function.

Description

Technical Field

[0001] The present disclosure relates to the technical field of speech translation, and in particular to a training method for a speech translation model, a speech translation method, a device, and a computer-readable medium. Background Art

[0002] Speech Translation (ST) aims to translate speech in a source language into text in a target language, and is widely used in scenarios such as multinational conferences and video subtitles. The speech translation system in the related art usually adopts a cascade approach, first using Automatic Speech Recognition (ASR) technology to convert the speech signal into a transcription in the source language, and then using Machine Translation (MT) technology to translate the transcription in the source language into a translation in the target language. However, the cascade approach usually has the disadvantages of error transmission and high latency. Error transmission refers to the fact that errors in speech recognition will be transmitted to the machine translation system, which may result in erroneous translations. The high latency is due to the fact that the source language speech to the target language text needs to pass through the ASR and MT models in succession, which increases the latency of the speech translation system to a certain extent.

[0003] In recent years, researchers have proposed end-to-end speech translation to solve the above two problems. End-to-end speech translation aims to use a model to complete the conversion from source language speech to target language text. Compared with text translation, speech translation tasks are more complex, but the data scale is usually smaller. In related technologies, technologies such as knowledge distillation can be used to improve the performance of speech translation models. Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] According to some embodiments of the present disclosure, a training method for a speech translation model is provided, comprising: obtaining a source speech representation sequence corresponding to source speech data and a source text representation sequence corresponding to the source speech data; obtaining a mixed sequence according to the source speech representation sequence and the source text representation sequence; processing the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and processing the mixed sequence using the speech translation model to output a second probability distribution of a second target text; calculating a total loss function according to the first probability distribution and the second probability distribution; and training the speech translation model according to the total loss function.

[0006] According to other embodiments of the present disclosure, a speech translation method is provided, comprising: collecting source speech data; and inputting the source speech data into a speech translation model to output a target text, wherein the speech translation model is trained by the training method of any embodiment described in the present disclosure.

[0007] According to some other embodiments of the present disclosure, a training device for a speech translation model is provided, comprising: a first acquisition unit, configured to obtain a source speech representation sequence corresponding to source speech data and a source text representation sequence corresponding to the source speech data; a second acquisition unit, configured to obtain a mixed sequence according to the source speech representation sequence and the source text representation sequence; a processing unit, configured to process the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and to process the mixed sequence using the speech translation model to output a second probability distribution of a second target text; a calculation unit, configured to calculate a total loss function according to the first probability distribution and the second probability distribution; and a training unit, configured to train the speech translation model according to the total loss function.

[0008] According to other embodiments of the present disclosure, a speech translation device is provided, comprising: a collection module for collecting source speech data; and an input module for inputting the source speech data into a speech translation model to output a target text, wherein the speech translation model is trained by the training method of any embodiment described in the present disclosure.

[0009] According to some other embodiments of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the electronic device executes the method of any embodiment described in the present disclosure.

[0010] According to some other embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of any embodiment described in the present disclosure is implemented.

[0011] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The drawings described herein are used to provide a further understanding of the present disclosure. Each of the drawings together with the following detailed description is included in this specification and forms a part of the specification, and is used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation on the present disclosure. In the drawings:

[0013] Figure 1 is a flow chart illustrating a training method for a speech translation model according to some embodiments of the present disclosure;

[0014] Figure 2 is a schematic diagram illustrating a training method for a speech translation model according to some embodiments of the present disclosure;

[0015] Figure 3 is a flowchart illustrating a speech translation method according to some embodiments of the present disclosure;

[0016] Figure 4 is a structural block diagram showing a training device for a speech translation model according to some embodiments of the present disclosure;

[0017] Figure 5 is a structural block diagram showing a speech translation device according to some embodiments of the present disclosure;

[0018] Figure 6 A block diagram showing an electronic device according to some embodiments of the present disclosure;

[0019] Figure 7 is a block diagram illustrating an example structure of a computer system that may be employed in some embodiments according to the present disclosure.

[0020] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure, but it is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of the embodiments is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein.

[0022] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0023] The term "include" and its variations used in the present disclosure mean an open term that includes at least the following elements / features but does not exclude other elements / features, that is, "including but not limited to". In addition, the term "include" and its variations used in the present disclosure mean an open term that includes at least the following elements / features but does not exclude other elements / features, that is, "including but not limited to". Therefore, include is synonymous with include. The term "based on" means "based at least in part on".

[0024] References throughout this specification to "one embodiment," "some embodiments," or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearances of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, but may do so.

[0025] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0029] The inventors of the present disclosure have found that the end-to-end speech translation model XSTNet or SATE in the related art uses large-scale machine translation data to pre-train the model, but when the model is connected to the front acoustic encoder (Acoustic Encoder), the performance usually decreases during the migration to speech translation. This is because there is a modality inconsistency between the acoustic representation (Acoustic Representation) encoded by the acoustic encoder and the text embedding (Text Embedding), which makes it difficult for the Transformer encoder-decoder, which has better performance in text translation, to process speech sequences.

[0030] In view of this, an embodiment of the present disclosure provides a training method for a speech translation model to improve the speech translation effect.

[0031] Figure 1 is a flowchart illustrating a training method for a speech translation model according to some embodiments of the present disclosure. Figure 2 is a schematic diagram showing a training method for a speech translation model according to some embodiments of the present disclosure. Figure 1 and Figure 2 A training method for a speech translation model according to some embodiments of the present disclosure is described in detail.

[0032] like Figure 1 As shown, in step S102, a source speech representation sequence corresponding to the source speech data and a source text representation sequence corresponding to the source speech data are obtained.

[0033] Here, the source speech representation sequence includes a speech vector corresponding to each word in the source speech data; the source text representation sequence includes a text vector corresponding to each word in the source speech data.

[0034] For example, Figure 2 As shown, the speech translation model may include an acoustic encoder, a translation encoder, and a translation decoder. For example, the acoustic encoder is a pre-trained wav2vec 2.0 model, and the translation encoder and translation decoder are Transformer models. The translation encoder can accept both the acoustic representation sequence s of the speech as input and the embedded representation sequence x of the transcribed text as input.

[0035] In other embodiments, Figure 2 As shown, the speech translation model may further include a feature extraction layer and an embedding layer. For example, the feature extraction layer may include CNN (Convolutional Neural Networks).

[0036] The following describes step S102 by taking the example of the speech translation model translating English speech into Chinese text.

[0037] For example, Figure 2 As shown in the figure, the source speech data is "I like sandwich". The acoustic encoder encodes the source speech data to obtain the corresponding vector. Then, the feature extraction layer extracts the vector to obtain the corresponding source speech representation sequence. The source speech representation sequence includes speech vectors corresponding to the words "I", "like", and "sandwich". For example, in Figure 2 In the figure, the white circle represents the speech vector of the word "I", the dotted circle represents the speech vector of the word "like", and the black circle represents the speech vector of the word "sandwich". In this way, the source speech representation sequence is obtained according to the source speech data.

[0038] Furthermore, the source text data "I like sand_wich" corresponding to the source speech data is processed by the embedding layer, etc., and the corresponding source text representation sequence can be obtained. The source text representation sequence includes the text vectors corresponding to the words "I", "like", and "sand_wich". For example, in Figure 2 In the figure, the white square represents the text vector of the word "I", the dotted square represents the text vector of the word "like", and the black square represents the text vector of the word "sand_wich". In this way, the source text representation sequence is obtained according to the source speech data.

[0039] Back to Figure 1, in step S104, a hybrid sequence (which can also be referred to as a cross-modal hybrid sequence) is obtained based on the source speech representation sequence and the source text representation sequence.

[0040] In some embodiments, step S104 includes: selecting one type of vector from the speech vectors and text vectors corresponding to each word in the source speech data as the vector corresponding to each word in the hybrid sequence; and forming the selected all vectors into a hybrid sequence. For example, randomly select one type of vector from the speech vectors and text vectors corresponding to each word in the source speech data as the vector corresponding to each word in the hybrid sequence.

[0041] For example, in the previous example, the source speech representation sequence includes the speech vectors corresponding to the words "I", "like", and "sandwich" respectively, and the source text representation sequence includes the text vectors corresponding to the words "I", "like", and "sand_wich" respectively. The vectors corresponding to the words "I", "like", and "sandwich" can be selected respectively by random selection. For example, as Figure 2 shown, the text vector of the word "I", the speech vector of the word "like", and the text vector of the word "sandwich" can be randomly selected to form a hybrid sequence.

[0042] It should be noted that the speech vector of each word can be a group of vectors, and the text vector of each word can also be a group of vectors. Therefore, when selecting one type of vector from the speech vector and text vector of a certain word, actually a group of speech vectors or a group of text vectors of that word is selected.

[0043] In some embodiments, the step of randomly selecting one type of vector from the speech vectors and text vectors corresponding to each word in the source speech data includes: generating a random number for each word in the source speech data; and comparing the random number with a preset threshold to select the vector corresponding to each word.

[0044] In some embodiments, the step of comparing the random number with a preset threshold to select the vector corresponding to each word includes: when the random number of a certain word is less than or equal to the threshold, selecting the speech vector of that certain word as the vector corresponding to that certain word in the hybrid sequence; or when the random number of a certain word is greater than the threshold, selecting the text vector of that certain word as the vector corresponding to that certain word in the hybrid sequence.

[0045] In some embodiments, the range of the threshold p is 0 < p < 1. Additionally, the range of the random number m generated for each word here is 0 < m < 1.

[0046] For example, as Figure 2As shown in the figure, the threshold p can be pre-set to 0.6. Generate random numbers for the words "I", "like", and "sandwich", for example, the random number m1 of the word "I" is 0.7, the random number m2 of the word "like" is 0.4, and the random number m3 of the word "sandwich" is 0.8. Since m1>0.6, the text vector of the word "I" is selected (by Figure 2 The white square in the figure is used as the vector of the word “I” in the mixed sequence; since m2<0.6, the speech vector of the word “like” is selected (represented by Figure 2 The dotted filled circles in the figure are used as the vector of the word “like” in the mixed sequence; since m3>0.6, the text vector of the word “sandwich” is selected (represented by Figure 2 The black square in the figure represents the vector of the word "sandwich" in the mixed sequence. In this way, after the mixing operation, the mixed sequence mixup((s,x),p) is formed, as shown in Figure 2 Here, s represents the speech sequence in the source language, and x represents the text sequence in the source language.

[0047] In the above embodiment, the paired <speech representation sequence, text representation sequence> are mixed at word level according to a preset threshold (or probability threshold), and the mixed sequence is used as the input of the translation encoder.<s,x> Perform forced alignment at the word level, where each word corresponds to a vector s in the speech representation sequence i , and the corresponding text represents the vector x in the sequence i According to the preset threshold p, each word randomly selects the corresponding speech representation sequence s i Or a text representation sequence x i , the concatenated sequence is the mixed sequence, denoted as mixup(<s,x> ,p).

[0048] Back to Figure 1 In step S106, the source speech representation sequence is processed using the speech translation model to output a first probability distribution of the first target text, and the mixed sequence is processed using the speech translation model to output a second probability distribution of the second target text.

[0049] For example, Figure 2As shown, the source speech representation sequence is processed by a translation encoder and a translation decoder to output a first probability distribution P(y|s) of a first target text, and the mixed sequence is processed by a speech translation model to output a second probability distribution P(y|mixup((s,x),p)) of a second target text, where y is a text sequence in the target language. Here, the translation encoder and the translation decoder in the speech translation model use known algorithms to process the input sequence, which will not be described in detail here.

[0050] In order to distinguish the first probability distribution from the second probability distribution, the first probability distribution can be recorded as P1(y1|s), and the second probability distribution can be recorded as P2(y2|mixup((s,x),p)), where y1 is the text sequence of the first target text in the target language, and y2 is the text sequence of the second target text in the target language. P1(y1|s) means that when the input is the speech sequence s, the model predicts the probability distribution of the translation y1, and P2(y2|mixup((s,x),p)) means that when the input is a mixed sequence, the model predicts the probability distribution of the translation y2.

[0051] Back to Figure 1 In step S108, a total loss function is calculated according to the first probability distribution and the second probability distribution.

[0052] In some embodiments, step S108 includes: calculating a first loss function based on the first probability distribution and the second probability distribution; calculating a KL divergence value (Kullback-Leible divergence, also known as relative entropy) based on the first probability distribution and the second probability distribution; calculating a second loss function based on the KL divergence value; and calculating a total loss function based on the first loss function and the second loss function.

[0053] In some embodiments, the first loss function is a weighted sum of a loss function corresponding to the first probability distribution and a loss function corresponding to the second probability distribution.

[0054] For example, the loss function corresponding to the first probability distribution and the loss function corresponding to the second probability distribution both use the known negative log-likelihood loss function (NLL for short), that is, the loss function NLL(). Then the loss function corresponding to the first probability distribution is NLL(P1(y1|s)), and the loss function corresponding to the second probability distribution is NLL(P2(y2|mixup((s,x),p))). Here, NLL(P1(y1|s)) means that the speech translation model calculates the negative log-likelihood loss function for the first probability distribution P1(y1|s), and NLL(P2(y2|mixup((s,x),p))) means that the speech translation model calculates the negative log-likelihood loss function for the second probability distribution P2(y2|mixup((s,x),p)). It should be noted that the negative log-likelihood loss function NLL() is a loss function known to those skilled in the art and will not be described in detail here.

[0055] Calculate the first loss function for

[0056]

[0057] here, The loss function is used to measure the distance between the first probability distribution P1(y1|s) and the second probability distribution P2(y2|mixup((s,x),p)) and the true probability distribution of the translation. α is the weight coefficient corresponding to the loss function NLL(P1(y1|s)), and β is the weight coefficient corresponding to the loss function NLL(P2(y2|mixup((s,x),p))). α and β are known coefficients and can be determined according to actual needs. For example, α>0, and β>0.

[0058] For example, when both α and β are 1, the above equation (1) is simplified to

[0059]

[0060] KL divergence value D KL It can measure the distance between two probability distributions.

[0061] In some embodiments, the KL divergence value includes: a forward KL divergence value and a backward KL divergence value. For example, for the first probability distribution P1(y1|s) and the second probability distribution P2(y2|mixup((s,x),p)), the forward KL divergence value D can be calculated. KL (P1||P2) and the backward KL divergence value D KL (P2||P1).

[0062] In some embodiments, the second loss function is the average of the forward KL divergence value and the backward KL divergence value. for

[0063]

[0064] It is a loss function that measures the distance between the first probability distribution P1(y1|s) and the second probability distribution P2(y2|mixup((s,x),p)).

[0065] Total loss function It can be used to update the model parameters to optimize the model. In some embodiments, the total loss function is the weighted sum of the first loss function and the second loss function. That is, the total loss function for

[0066]

[0067] Among them, γ is the first loss function The corresponding weight coefficient, δ, is related to the second loss function The corresponding weight coefficients. γ and δ are known coefficients and can be determined according to actual needs. For example, γ>0, and δ≥0.

[0068] For example, when both γ and δ are 1, the above equation (4) is simplified to

[0069]

[0070] So far, the calculation methods of some embodiments of the total loss function have been implemented.

[0071] In the calculation process of the above total loss function, in addition to calculating the loss functions corresponding to the two probability distributions, the KL divergence value is also introduced to measure the distance between the two probability distributions, which can better narrow the distance between the two probability distributions and further improve the performance of speech translation. The above total loss function can be used to optimize the parameters more accurately and quickly in the subsequent process of optimizing model parameters.

[0072] It should be noted that the calculation method of the total loss function is not limited to the above calculation method. For example, the above first loss function can be used as the total loss function without introducing the above KL divergence value.

[0073] In step S110, a speech translation model is trained according to the total loss function.

[0074] That is, the parameters in the speech translation model are updated according to the total loss function calculated in step S108.

[0075] So far, a training method for a speech translation model according to some embodiments of the present disclosure is provided. The training method includes: obtaining a source speech representation sequence corresponding to the source speech data and a source text representation sequence corresponding to the source speech data; obtaining a mixed sequence according to the source speech representation sequence and the source text representation sequence; processing the source speech representation sequence using a speech translation model to output a first probability distribution of a first target text, and processing the mixed sequence using a speech translation model to output a second probability distribution of a second target text; calculating a total loss function according to the first probability distribution and the second probability distribution; and training the speech translation model according to the total loss function. In the training method, a multi-task learning method is adopted, and in addition to training the model using a speech representation sequence, the model is also trained using a mixed sequence. Since the mixed model contains a speech vector component and a text vector component, this can implicitly narrow the distance between the speech representation and the text representation, thereby assisting the speech sequence training, making the speech sequence training simpler and more accurate. When the speech translation model trained by such a training method is used for speech translation, the speech translation effect can be improved.

[0076] It should be noted that although the above example is a translation from English speech to Chinese text, the model and method of the present disclosure are not limited to this, and the model and method of the present disclosure can be applied to translation between any languages.

[0077] It should also be noted that the term "word" used in the present disclosure may not be understood as just a character or a word. For example, in English, a "word" may be understood as a word, or a word combination, or a phrase, etc.; in Chinese, a "word" may be understood as a character, or a phrase, or a phrase, etc., and other languages ​​are similar. Therefore, the meaning of "word" in the present disclosure may be determined according to actual needs.

[0078] In some embodiments, the training method may further include: before step S102, pre-training the speech translation model using parallel corpus of machine translation.

[0079] In some embodiments of the present disclosure, a large-scale parallel corpus of machine translation may be used for pre-training first, and then the speech translation model may be trained by fine-tuning. In the fine-tuning stage, a multi-task learning method is used to train two tasks simultaneously: the first task is a conventional speech translation task, and the second task is a translation task when the input is a cross-modal mixed sequence. The two tasks are trained simultaneously.

[0080] Figure 3 is a flow chart showing a method for speech translation according to some embodiments of the present disclosure. Figure 3 As shown, the speech translation method includes steps S302 to S304.

[0081] In step S302, source voice data is collected.

[0082] In step S304, the source speech data is input into the speech translation model to output the target text. Here, the speech translation model is trained by the training method as described above.

[0083] So far, a speech translation method according to some embodiments of the present disclosure is provided. Since in this method, a speech translation model trained by the training method as described above is used to translate the source speech data, the speech translation effect can be improved.

[0084] The embodiment of the present disclosure uses the MuST-C dataset (English-German) in speech translation to test the performance of the model. The comparison with other speech translation models is shown in Table 1. Table 1 uses the BLEU (Bilingual Evaluation Understudy) indicator commonly used in machine translation. It can be seen that the speech translation model of the present disclosure has achieved the best effect in end-to-end speech translation.

[0085] Table 1 Comparison of BLEU scores of the public model and other models on the MuST-C En-De data test set

[0086]

[0087] Figure 4 is a structural block diagram showing a training device for a speech translation model according to some embodiments of the present disclosure. Figure 4 As shown, the training device includes a first acquisition unit 402 , a second acquisition unit 404 , a processing unit 406 , a calculation unit 408 and a training unit 410 .

[0088] The first acquisition unit 402 is used to obtain a source speech representation sequence corresponding to the source speech data and a source text representation sequence corresponding to the source speech data.

[0089] In some embodiments, the source speech representation sequence includes a speech vector corresponding to each word in the source speech data, and the source text representation sequence includes a text vector corresponding to each word in the source speech data.

[0090] The second acquisition unit 404 is used to obtain a mixed sequence according to the source speech representation sequence and the source text representation sequence.

[0091] In some embodiments, the second acquisition unit 404 is used to select a vector from the speech vector and the text vector corresponding to each word in the source speech data as the vector corresponding to each word in the mixed sequence, and to form a mixed sequence with all the selected vectors.

[0092] For example, the second acquisition unit 404 is used to randomly select a vector from the voice vector and the text vector corresponding to each word in the source voice data as the vector corresponding to each word in the mixed sequence.

[0093] In some embodiments, the second acquisition unit 404 is used to generate a random number for each word in the source speech data, and compare the random number with a preset threshold to select a vector corresponding to each word.

[0094] For example, the second acquisition unit 404 is used to select the speech vector of a certain word as the vector corresponding to the certain word in the mixed sequence when the random number of the certain word is less than or equal to a threshold, or to select the text vector of the certain word as the vector corresponding to the certain word in the mixed sequence when the random number of the certain word is greater than the threshold.

[0095] The processing unit 406 is used to process the source speech representation sequence using the speech translation model to output a first probability distribution of the first target text, and to process the mixed sequence using the speech translation model to output a second probability distribution of the second target text.

[0096] The calculation unit 408 is used to calculate the total loss function according to the first probability distribution and the second probability distribution.

[0097] In some embodiments, the calculation unit 408 is used to calculate a first loss function according to the first probability distribution and the second probability distribution, calculate a KL divergence value according to the first probability distribution and the second probability distribution, calculate a second loss function according to the KL divergence value, and calculate a total loss function according to the first loss function and the second loss function.

[0098] In some embodiments, the first loss function is a weighted sum of a loss function corresponding to the first probability distribution and a loss function corresponding to the second probability distribution.

[0099] In some embodiments, the KL divergence value includes: a forward KL divergence value and a backward KL divergence value.

[0100] In some embodiments, the second loss function is an average of the forward KL divergence value and the backward KL divergence value.

[0101] In some embodiments, the total loss function is a weighted sum of the first loss function and the second loss function.

[0102] The training unit 410 is used to train the speech translation model according to the total loss function.

[0103] So far, a training device for a speech translation model according to some embodiments of the present disclosure has been described. The training device includes a first acquisition unit, a second acquisition unit, a processing unit, a computing unit, and a training unit. In the training device, a multi-task learning method is adopted. In addition to using a speech representation sequence to train the model, a mixed sequence is also used to train the model. Since the mixed model contains a speech vector component and a text vector component, this can implicitly narrow the distance between the speech representation and the text representation, thereby assisting the speech sequence training, making the speech sequence training simpler and more accurate. When performing speech translation using a speech translation model trained by such a training device, the speech translation effect can be improved.

[0104] In some embodiments, the training device further includes a pre-training unit, which is used to pre-train the speech translation model using parallel corpus for machine translation.

[0105] Figure 5 is a structural block diagram showing a speech translation device according to some embodiments of the present disclosure. Figure 5 As shown, the speech translation device includes a collection module 502 and an input module 504 .

[0106] The collection module 502 is used to collect source voice data.

[0107] The input module 504 is used to input the source speech data into the speech translation model to output the target text. The speech translation model is trained by the training method as described above (e.g. Figure 1 The training method shown in the figure is used for training.

[0108] So far, a speech translation device according to some embodiments of the present disclosure is provided. Since the speech translation device uses a speech translation model trained by the training method as described above to translate the source speech data, the speech translation effect can be improved.

[0109] It should be noted that the above-mentioned units are only logical modules divided according to the specific functions implemented by them, and are not used to limit the specific implementation methods. For example, they can be implemented in software, hardware, or a combination of software and hardware. In actual implementation, the above-mentioned units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). In addition, the operations / functions implemented by the above-mentioned units can be implemented by the processing circuit itself.

[0110] In addition, although not shown, the device may also include a memory, which may store various information generated by the device, various units contained in the device during operation, programs and data for operation, data to be sent by the communication unit, etc. The memory may be a volatile memory and / or a non-volatile memory. For example, the memory may include, but is not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), and a flash memory. Of course, the memory may also be located outside the device. Optionally, although not shown, the device may also include a communication unit, which may be used to communicate with other devices. In one example, the communication unit may be implemented in a suitable manner known in the art, such as including communication components such as an antenna array and / or a radio frequency link, various types of interfaces, communication units, etc. It will not be described in detail here. In addition, the device may also include other components not shown, such as a radio frequency link, a baseband processing unit, a network interface, a processor, a controller, etc. It will not be described in detail here.

[0111] Some embodiments of the present disclosure also provide an electronic device. Figure 6 A block diagram of an electronic device according to some embodiments of the present disclosure is shown. For example, in some embodiments, the electronic device 6 can be various types of devices, for example, can include but are not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. For example, the electronic device 6 can include a display panel for displaying data and / or execution results used in the scheme according to the present disclosure. For example, the display panel can be in various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel, etc. In addition, the display panel can be not only a flat panel, but also a curved panel, or even a spherical panel.

[0112] like Figure 6 As shown, the electronic device 6 of this embodiment includes: a memory 61 and a processor 62 coupled to the memory 61. It should be noted that Figure 6 The components of the electronic device 6 shown are only exemplary and non-restrictive. The electronic device 6 may also have other components according to actual application requirements. The processor 62 may control other components in the electronic device 6 to perform desired functions.

[0113] In some embodiments, the memory 61 is used to store one or more computer-readable instructions. When the processor 62 is used to execute the computer-readable instructions, the computer-readable instructions are executed by the processor 62 to implement the method according to any of the above embodiments. For the specific implementation of each step of the method and the related explanation content, please refer to the above embodiments, and the repeated parts will not be repeated here.

[0114] For example, the processor 62 and the memory 61 may communicate with each other directly or indirectly. For example, the processor 62 and the memory 61 may communicate with each other through a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 62 and the memory 61 may also communicate with each other through a system bus, which is not limited by the present disclosure.

[0115] For example, the processor 62 may be embodied as various appropriate processors, processing devices, etc., such as a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X86 or ARM architecture, etc. For example, the memory 61 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The memory 61 may include, for example, a system memory, which may store, for example, an operating system, an application, a boot loader (Boot Loader), a database, and other programs. Various applications and various data may also be stored in the storage medium.

[0116] In addition, according to some embodiments of the present disclosure, various operations / processes according to the present disclosure, when implemented by software and / or firmware, can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 7 The computer system 700 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 7 is a block diagram illustrating an example structure of a computer system that may be employed in some embodiments according to the present disclosure.

[0117] exist Figure 7In the embodiment, the central processing unit (CPU) 701 performs various processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703. In the RAM 703, data required when the CPU 701 performs various processes, etc. are also stored as needed. The central processing unit is only exemplary, and it can also be other types of processors, such as the various processors described above. The ROM 702, RAM 703 and the storage part 708 can be various forms of computer-readable storage media, as described below. It should be noted that although Figure 7 ROM 702, RAM 703 and storage section 708 are shown separately in FIG. 7 , but one or more of them may be combined or located in the same or different memory or storage modules.

[0118] The CPU 701, the ROM 702, and the RAM 703 are connected to one another via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0119] The following components are connected to the input / output interface 705: an input portion 706, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output portion 707, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage portion 708, including a hard disk, a magnetic tape, etc.; and a communication portion 709, including a network interface card such as a LAN card, a modem, etc. The communication portion 709 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 7 The various devices or modules in the electronic device 700 are shown to communicate via the bus 704, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network.

[0120] A drive 710 is also connected to the input / output interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is mounted on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0121] In the case where the above-described series of processing is implemented by software, a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 711 .

[0122] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, or installed from the storage part 708, or installed from the ROM 702. When the computer program is executed by the CPU 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0123] It should be noted that in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. For example, a computer-readable storage medium is a non-transitory computer-readable storage medium. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer-readable storage medium may be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that may be used by an instruction execution system, device or device or used in combination with it. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0124] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0125] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute any of the methods of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0126] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0129] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0130] According to some embodiments of the present disclosure, a training method for a speech translation model is provided, comprising: obtaining a source speech representation sequence corresponding to source speech data and a source text representation sequence corresponding to the source speech data; obtaining a mixed sequence according to the source speech representation sequence and the source text representation sequence; processing the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and processing the mixed sequence using the speech translation model to output a second probability distribution of a second target text; calculating a total loss function according to the first probability distribution and the second probability distribution; and training the speech translation model according to the total loss function.

[0131] In some embodiments, the step of calculating the total loss function according to the first probability distribution and the second probability distribution includes: calculating a first loss function according to the first probability distribution and the second probability distribution; calculating a KL divergence value according to the first probability distribution and the second probability distribution; calculating a second loss function according to the KL divergence value; and calculating a total loss function according to the first loss function and the second loss function.

[0132] In some embodiments, the KL divergence value includes: a forward KL divergence value and a backward KL divergence value.

[0133] In some embodiments, the second loss function is an average of the forward KL divergence value and the backward KL divergence value.

[0134] In some embodiments, the total loss function is a weighted sum of the first loss function and the second loss function.

[0135] In some embodiments, the first loss function is a weighted sum of a loss function corresponding to the first probability distribution and a loss function corresponding to the second probability distribution.

[0136] In some embodiments, the source speech representation sequence includes a speech vector corresponding to each word in the source speech data; and the source text representation sequence includes a text vector corresponding to each word in the source speech data.

[0137] In some embodiments, the step of obtaining the mixed sequence according to the source speech representation sequence and the source text representation sequence includes: selecting a vector from the speech vector and the text vector corresponding to each word in the source speech data as the vector corresponding to each word in the mixed sequence; and forming the mixed sequence with all the selected vectors.

[0138] In some embodiments, a vector is randomly selected from the speech vector and the text vector corresponding to each word in the source speech data as the vector corresponding to each word in the mixed sequence.

[0139] In some embodiments, the step of randomly selecting a vector from the speech vector and the text vector corresponding to each word in the source speech data includes: generating a random number for each word in the source speech data; and comparing the random number with a preset threshold to select a vector corresponding to each word.

[0140] In some embodiments, the step of comparing the random number with a preset threshold to select a vector corresponding to each word includes: when the random number of a certain word is less than or equal to the threshold, selecting the speech vector of the certain word as the vector corresponding to the certain word in the mixed sequence; or when the random number of a certain word is greater than the threshold, selecting the text vector of the certain word as the vector corresponding to the certain word in the mixed sequence.

[0141] In some embodiments, the training method further comprises: before obtaining the source speech representation sequence and the source text representation sequence, pre-training the speech translation model using parallel corpus of machine translation.

[0142] According to some other embodiments of the present disclosure, a speech translation method is provided, comprising: collecting source speech data; and inputting the source speech data into a speech translation model to output a target text, wherein the speech translation model is trained by the training method as described above.

[0143] According to some other embodiments of the present disclosure, a training device for a speech translation model is provided, comprising: a first acquisition unit, configured to obtain a source speech representation sequence corresponding to source speech data and a source text representation sequence corresponding to the source speech data; a second acquisition unit, configured to obtain a mixed sequence according to the source speech representation sequence and the source text representation sequence; a processing unit, configured to process the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and to process the mixed sequence using the speech translation model to output a second probability distribution of a second target text; a calculation unit, configured to calculate a total loss function according to the first probability distribution and the second probability distribution; and a training unit, configured to train the speech translation model according to the total loss function.

[0144] According to some other embodiments of the present disclosure, a speech translation device is provided, comprising: a collection module for collecting source speech data; and an input module for inputting the source speech data into a speech translation model to output a target text, wherein the speech translation model is trained by the training method as described above.

[0145] According to some other embodiments of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the electronic device executes the method of any embodiment described in the present disclosure.

[0146] According to some other embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method of any embodiment described in the present disclosure is implemented.

[0147] According to still other embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method of any one of the embodiments described in the present disclosure.

[0148] According to some embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, implement the method of any embodiment described in the present disclosure.

[0149] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0150] In the description provided herein, many specific details are set forth. However, it is understood that embodiments of the present invention can be implemented without these specific details. In other cases, well-known methods, structures, and techniques are not presented in detail in order not to obscure the understanding of the description.

[0151] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0152] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A training method for a speech translation model, comprising: Obtaining a source speech representation sequence corresponding to the source speech data and a source text representation sequence corresponding to the source speech data; Obtaining a mixed sequence according to the source speech representation sequence and the source text representation sequence; Processing the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and processing the mixed sequence using the speech translation model to output a second probability distribution of a second target text; Calculate a total loss function according to the first probability distribution and the second probability distribution, wherein the step of calculating the total loss function according to the first probability distribution and the second probability distribution comprises: calculating a first loss function according to the first probability distribution and the second probability distribution; calculating a Kulbeck-Leibler KL divergence value according to the first probability distribution and the second probability distribution; calculating a second loss function according to the KL divergence value; and calculating a total loss function according to the first loss function and the second loss function; and The speech translation model is trained according to the total loss function.

2. The training method according to claim 1, wherein: The KL divergence value includes: a forward KL divergence value and a backward KL divergence value.

3. The training method for a speech translation model according to claim 2, wherein: The second loss function is an average of the forward KL divergence value and the backward KL divergence value.

4. The training method according to claim 1, wherein: The total loss function is a weighted sum of the first loss function and the second loss function.

5. The training method according to claim 1, wherein: The first loss function is a weighted sum of a loss function corresponding to the first probability distribution and a loss function corresponding to the second probability distribution.

6. The training method according to claim 1, wherein: The source speech representation sequence includes a speech vector corresponding to each word in the source speech data; The source text representation sequence includes a text vector corresponding to each word in the source speech data.

7. The training method according to claim 6, wherein: The step of obtaining the mixed sequence according to the source speech representation sequence and the source text representation sequence comprises: Selecting a vector from the voice vector and the text vector corresponding to each word in the source voice data as the vector corresponding to each word in the mixed sequence; and All the selected vectors are combined into the mixed sequence.

8. The training method according to claim 7, wherein: A vector is randomly selected from the speech vector and the text vector corresponding to each word in the source speech data as the vector corresponding to each word in the mixed sequence.

9. The training method according to claim 8, wherein: The step of randomly selecting a vector from the voice vector and the text vector corresponding to each word in the source voice data comprises: generating a random number for each word in the source speech data; and The random number is compared with a preset threshold to select a vector corresponding to each word.

10. The training method according to claim 9, wherein: The step of comparing the random number with a preset threshold to select a vector corresponding to each word comprises: When the random number of a certain word is less than or equal to the threshold, the speech vector of the certain word is selected as the vector corresponding to the certain word in the mixed sequence; or When the random number of a certain word is greater than the threshold, the text vector of the certain word is selected as the vector corresponding to the certain word in the mixed sequence.

11. The training method according to claim 1, further comprising: Before obtaining the source speech representation sequence and the source text representation sequence, the speech translation model is pre-trained using parallel corpus of machine translation.

12. A speech translation method, comprising: Collect source voice data; and The source speech data is input into a speech translation model to output a target text, wherein the speech translation model is trained by the training method according to any one of claims 1 to 11.

13. A training device for a speech translation model, comprising: A first acquisition unit, configured to obtain a source speech representation sequence corresponding to the source speech data and a source text representation sequence corresponding to the source speech data; A second acquisition unit, configured to obtain a mixed sequence according to the source speech representation sequence and the source text representation sequence; a processing unit, configured to process the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and to process the mixed sequence using the speech translation model to output a second probability distribution of a second target text; a calculation unit, configured to calculate a total loss function according to the first probability distribution and the second probability distribution, wherein the calculation unit is configured to calculate a first loss function according to the first probability distribution and the second probability distribution, calculate a Kulbeck-Leibler KL divergence value according to the first probability distribution and the second probability distribution, calculate a second loss function according to the KL divergence value, and calculate a total loss function according to the first loss function and the second loss function; and A training unit is used to train the speech translation model according to the total loss function.

14. A speech translation device, comprising: A collection module, used for collecting source voice data; and An input module is used to input the source speech data into a speech translation model to output a target text, wherein the speech translation model is trained by the training method according to any one of claims 1 to 11.

15. An electronic device, comprising: Memory; and A processor coupled to the memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 12.

16. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images