Method, apparatus, electronic device and medium for speech translation

Abstract

Embodiments of the present disclosure relate to a method, apparatus, electronic device and medium for speech translation. The method comprises obtaining a plurality of audios in a source language, and generating mixed speech features by multi-level augmentation on the plurality of audios, wherein the multi-level comprises at least two of a word level, a sentence level, or a frame level. The method further comprises generating target language texts corresponding to the plurality of audios based on the mixed speech features. According to embodiments of the present disclosure, limited speech translation data can be more fully utilized, data augmentation is achieved by multi-level augmentation, and thus the accuracy of speech translation is improved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more specifically, to methods, apparatus, electronic devices, and media for speech translation. Background Technology

[0002] Speech translation is a cross-modal task from speech to text, aiming to translate speech in one language into text in another language, and has a wide range of applications. Traditional speech translation methods mainly consist of a cascaded speech recognition model and a machine translation model. The speech recognition model converts the input speech of the source language into text of the source language, and the machine translation model translates the text of the source language output by the speech recognition model into text of the target language.

[0003] Errors may occur when translating speech from the source language to text in the source language. These errors may then be further amplified during the translation from source language text to target language text, or they may be related to errors in words, grammar, etc. Furthermore, the cascading of speech recognition and machine translation models also contributes to significant latency in speech translation tasks, meaning the processing time between inputting speech in the source language and outputting text in the target language is relatively long. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, and computer-readable storage medium for speech translation.

[0005] According to a first aspect of this disclosure, a method for speech translation is provided. The method includes acquiring multiple audio recordings of a source language. The method further includes generating mixed speech features by multi-level amplification of the multiple audio recordings, wherein the multi-level amplification includes at least two of the word level, sentence level, or frame level. The method also includes generating target language text corresponding to the multiple audio recordings based on the mixed speech features.

[0006] In a second aspect of this disclosure, an apparatus for speech translation is provided. The apparatus includes a speech acquisition module configured to acquire multiple audio samples of a source language. The apparatus also includes a multi-level augmentation module configured to generate mixed speech features through multi-level augmentation of the multiple audio samples, wherein the multi-level augmentation includes at least two of the word level, sentence level, or frame level. The apparatus further includes a text generation module configured to generate target language text corresponding to the multiple audio samples based on the mixed speech features.

[0007] According to a third aspect of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to the first aspect.

[0008] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.

[0009] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which a speech translation method according to some embodiments of the present disclosure may be implemented is shown;

[0012] Figure 2 A flowchart of a speech translation method according to some embodiments of the present disclosure is shown;

[0013] Figure 3 A schematic diagram of a multi-level amplification process according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram illustrating the process of training a speech translation model according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram illustrating the process of fine-tuning a speech translation model according to some embodiments of the present disclosure is shown;

[0016] Figure 6 A schematic diagram of an example architecture of a speech translation model according to some embodiments of the present disclosure is shown;

[0017] Figure 7 A block diagram of an apparatus for speech translation according to some embodiments of the present disclosure is shown; and

[0018] Figure 8 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0019] In all the accompanying figures, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the type, scope of use, and usage scenarios of the personal information (such as voice) involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information (such as voice) to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosure. It is understood that the above notification and user authorization process is merely illustrative and does not limit the implementation of this disclosure; other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0025] In some embodiments of this disclosure, a speech translation task from English speech to German text will be described as an example; however, "speech" and "text" of other languages ​​may also be used in conjunction with embodiments of this disclosure. Furthermore, all specific numerical values ​​herein are illustrative and are intended to aid understanding only, and are not intended to limit the scope of the invention.

[0026] As discussed above, cascaded models of traditional speech recognition and machine translation can be trained on datasets from both speech recognition and machine translation tasks. These datasets are large in scale and readily available. Machine translation is also widely used because obtaining large amounts of machine translation data is also easy. However, traditional cascaded models suffer from some issues regarding accuracy and latency.

[0027] However, research has shown that end-to-end speech translation models can overcome the shortcomings of cascaded methods and surpass the translation performance of cascaded models. However, training end-to-end speech translation models faces the problem of data scarcity, as the datasets for speech translation tasks are much smaller than those for speech recognition and machine translation. Existing solutions utilize machine translation task datasets to assist in the training of speech translation models.

[0028] To address the aforementioned shortcomings, embodiments of this disclosure provide a speech translation scheme. This scheme can utilize existing training data to achieve data augmentation, effectively solving the data scarcity problem. The scheme provides a multi-level data augmentation method that can augment the original training data in different dimensions, resulting in augmented training data containing varying degrees of semantic or timbre mixing. This allows the speech translation model to learn more hidden features, providing more comprehensive utilization of limited training data and thereby improving the accuracy of speech translation.

[0029] In the following description, some embodiments will be discussed with reference to the process of speech translation between English speech and German text. However, it should be understood that this is merely to enable those skilled in the art to better understand the principles and ideas of the embodiments of this disclosure, and is not intended to limit the scope of this disclosure in any way.

[0030] Figure 1 A schematic diagram of an example environment 100 in which a speech translation method according to certain embodiments of the present disclosure may be implemented is shown. Figure 1 As shown, example environment 100 may include computing device 110, which may be a user terminal, mobile device, computer, etc., or it may be a computing system, a single server, a distributed server, or a cloud-based server. Computing device 110 may receive audio 160-1, audio 160-2, audio 160-3, and audio 160-4 (collectively referred to individually or collectively as audio 160). Audio 160 can be understood as a duration of speech, such as the sound of a user speaking, which contains meaningful content and can typically be transcribed into text. It is understood that environment 100 may also include more or fewer audio devices.

[0031] The computing device 110 may also include a speech translation system 120. For example, the speech translation system 120 may be deployed in the computing device 110. The speech translation system 120 may be used to generate a translation result of audio 160, namely text 170-1 or text 170-2 (referred to individually or collectively as text 170). In some embodiments, the speech translation system 120 may be trained based on a machine learning model architecture by utilizing a loss function associated with the acoustic properties of the audio and the labeled corresponding text.

[0032] refer to Figure 1 The speech translation system 120 includes a multi-level augmentation model 130. Using the multi-level augmentation model 130, more audio training data can be obtained from the original audio training data using a multi-level augmentation expansion method for training. This allows the machine learning model to learn optimized model parameters and obtain a trained model for use in the inference stage.

[0033] The speech translation system 120 also includes an acoustic encoder 140. The acoustic encoder 140 can convert the speech features of the original audio and the amplified audio into a word vector matrix. The word vector matrix is ​​then input into the speech translation model 150. The speech translation model 150 will determine multiple probabilities for each word in the text corresponding to the audio 160, and determine the text 170 based on these probabilities.

[0034] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0035] The following will combine Figures 2 to 6 The process according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0036] Figure 2 A flowchart of a speech translation method 200 according to certain embodiments of the present disclosure is shown. Method 200 can be used when training a speech translation system 120. At block 202, multiple audio recordings of the source language are acquired. For example, the speech translation system 120 acquires audio recordings 160-1 and 160-2, or audio recordings 160-1, 160-2, 160-3, and 160-4.

[0037] At box 204, a mixed speech feature is generated through multi-level amplification of multiple audio samples. The multi-level amplification includes at least two of the following: word level, sentence level, or frame level. It can be understood that this mixed speech feature can represent the acoustic characteristics of the audio in the form of a high-dimensional vector (for example, 64-dimensional, 128-dimensional, 256-dimensional, etc.), such as features representing the concentration of energy regions in timbre, and features representing prosodic characteristics such as duration, fundamental frequency, and average speech power. The following section combines... Figure 3 and Figure 4 An example implementation of how to achieve multi-level amplification is described.

[0038] At box 206, target language text corresponding to multiple audio recordings is generated based on the mixed speech features. As an example, the mixed speech features are converted into word vectors by an acoustic encoder, and then the corresponding text is generated through a speech translation model. This corresponding text is already in the target language, thus achieving end-to-end speech translation from source language speech to target language text.

[0039] In this way, the original training data can be augmented in multiple dimensions, so that the augmented training data includes a mixture of different degrees of semantics or timbre. This allows the speech translation model to learn more hidden features, thus making fuller use of the limited training data and improving the accuracy of speech translation.

[0040] Figure 3 A schematic diagram of a multi-level amplification process 300 according to some embodiments of the present disclosure is shown. Process 300 can be performed in... Figure 1 This is implemented in the multi-level augmentation model 130. In process 300, the corpus can be augmented in multiple dimensions based on the acquired audio 160, making fuller use of the limited training data.

[0041] At word level 302, audio can be amplified at the word level. As an example, the acquired audio S1 (also known as the first audio): So I told them this story and audio S2 (also known as the second audio): It's just cash can be amplified at the word level.

[0042] In some embodiments, word-level augmentation can be achieved based on triple substitution. For example, training data can be represented as triples of (speech, transcribed text, translation). Speech represents the audio of the source language, transcribed text represents the text corresponding to the speech in the source language, and translated text represents the text in the target language. For example, for the training corpus triples D = {(s, x, y)}, where D represents a triple and S = [s1, s2, ..., s...],... |s|[x1, x2, ..., x] represents the audio of the source language; X = [x1, x2, ..., x] |y| [] represents the text corresponding to the pinyin of the source language (i.e., the transcribed text); Y = [y1, y2, ..., y |y| [] represents text in the target language.

[0043] Therefore, we can first replace the transcribed text representing "story" in the triplet of S1 with its synonym "article". Then, we combine the pronunciation of "article" with the unreplaced part of the speech to form a new audio S1', i.e., "So I told them this article". Similarly, we can first replace the transcribed text representing "cash" in the triplet of S2 with its synonym "money". Then, we combine the pronunciation of "money" with the unreplaced part of the speech to form a new audio S2', i.e., "It's just money".

[0044] Similarly, the acquired audio segments S3 (also known as the third audio): So that was the book and S4 (also known as the fourth audio): That's my wish can be expanded at the word level. For example, replacing "book" with the synonym "notebook" gives S3': So that was the notebook. For S4, replacing "wish" with the synonym "hope" gives S4': That's my hope.

[0045] Next, at sentence level 304, two sentences can be concatenated. For example, at concatenation 306-1, S1' and S2' are concatenated, resulting in audio S5 where S1' and S2' are joined end-to-end: So I told them this article It's just money. At concatenation 306-2, S3' and S4' are concatenated, resulting in audio S6 where S3' and S4' are joined end-to-end: So that was the notebook That's my hope. Audio S5 is also known as first-word-level amplified audio, and audio S6 is also known as second-word-level amplified audio.

[0046] In some embodiments, the audio used for word-level amplification may be randomly selected. Alternatively, the audio used for word-level amplification may be arranged in the order of acquisition. According to embodiments of this disclosure, it is desirable that the amplified data can maximize the accuracy and generalization ability of the speech translation system 120.

[0047] At frame level 308, frame-level augmentation can be performed on audio S5 and audio S6. In some embodiments, frame-level augmentation can weight the speech features of audio S5 and audio S6. For example, at mixing 310, S5 can be weighted with p* and S6 can be weighted with 1-p*. The weighted mixed speech features 312 are output to the acoustic encoder 140 for further processing. For example, as expressed by formula (1):

[0048] s mix =p*s i +(1-p*)s j (1)

[0049] Among them, s mix Represents a sequence of mixed speech; s i s j These represent the amplified audio at the sentence level; i and j represent the audio counts.

[0050] Figure 4 A schematic diagram of a process 400 for training a speech translation model according to some embodiments of the present disclosure is shown. Figure 4 In the middle, the acoustic encoder 404 can be Figure 1 The acoustic encoder 140 in the middle, and the speech translation model 410 can be Figure 1 The speech translation model 150 in the text.

[0051] The acoustic encoder 404 may include a language model layer 408 (e.g., a Hubert layer, a bidirectional encoder representation model with hidden units) and a convolutional neural network (CNN) layer 406. The Hubert layer 408 learns self-supervised speech representations through hidden unit mask prediction. As an example, the Hubert layer 408 determines a masked region of the audio based on the mixed speech features 312. Based on this masked region, its hidden units are determined, and based on these hidden units, word vectors for the audio are generated.

[0052] The prediction loss of the Hubert layer 408 during prediction can be applied to masked regions, leveraging good high-level representations of the unmasked input to correctly infer the target of the masked input. The Hubert layer 408 learns a combined acoustic and language model on continuous inputs, thus generating high-quality word vectors and increasing the translation accuracy of subsequent speech translation models.

[0053] In some embodiments, the dimensionality of the word vectors can be further reduced using CNN layer 406. As an example, it can be reduced from 256 dimensions to the desired target dimension, such as 128 dimensions. This reduces memory requirements and can be adjusted as needed, for example, to balance computational overhead, processing speed, and translation accuracy.

[0054] During training 400, the acoustic encoder 404 also acquires unmixed speech features, such as the speech features of audio S5 and audio S6, as data used during training. When training the speech translation model 410, it is also necessary to use the text corresponding to the audio to enable the speech translation model 410 to learn the relevant speech features and adjust the model's parameters.

[0055] In some embodiments, process 400 may prepare the German text Y1 corresponding to the audio S1: also er Ich ihnen diese geschichte (So I told them this story), and the German text Y2 corresponding to audio S2: es ist einfach bargeld (It's just cash). In German text Y1: replacing geschichte with its synonym artikel (article), we get German text Y1': also er ich ihnen dieseartikel. In the German text Y2: replacing bargeld with its synonym geld (money) gives the German text Y2': es isteinfach geld.

[0056] Process 400 can prepare the German text Y3 corresponding to audio S3: darum ginges also in diesembuch (This is the book), and the German text Y4 corresponding to audio S4: das ist mein wunsch (That is my wish). In German text Y3: replace buch with its synonym notizbuch (notebook), resulting in German text Y3': darum ginges also in diesem notizbuch. In German text Y4: replace wunsch with its synonym hoffnung (hope), resulting in German text Y4': das ist mein hoffnung.

[0057] At concatenation point 430, the German texts Y1' and Y2' can be concatenated to obtain the German text Y5: alsoer. At splicing point 432, German texts Y3' and Y4' can be spliced ​​together to obtain German text Y6: darum ginges also in diesem notizbuchdas ist mein hoffnung.

[0058] In this way, the corresponding labeled text can be prepared. It can be seen that English audio S1 corresponds to German text Y1, English audio S2 corresponds to German text Y2, and so on. During training, unmixed speech features 402 (e.g., audio S5 and S6) are input into the speech translation model 410. Mixed speech features 312 are also input into the speech translation model 410. The speech translation model 410 translates the unmixed speech features 402 and the mixed speech features 312 into German text via a translation encoder 414 and a translation decoder 412, respectively. It should be noted that this disclosure uses only four audio samples as an example, but in practice, there can be more audio data; therefore, the translated German text will be represented by distributions 416 and 418. Distribution 416 is associated with the unmixed speech features 402, and distribution 418 is associated with the mixed speech features 312. Distributions 416 and 418 are represented in the form of probability distributions.

[0059] Thus, three loss functions can be calculated separately: L1 (also known as the first loss function) for the output of audio S5 and text Y5, and L2 (also known as the second loss function) for the output of audio S6 and text Y6. It can be understood that the weights 420 of loss functions L1 and L2 are both 1. A loss function L3 can also be calculated to correlate the mixed audio features 312 with text Y5 and text Y6. Here, loss function L1 has a weight of p* (422), loss function L2 has a weight of 1-p* (424), and the weighted average of loss functions L1 and L2 yields loss function L3 (also known as the third loss function).

[0060] As an example, the first loss function can be calculated using the following formula (2):

[0061] L i = L CE (s mix , y i (2)

[0062] Where i represents the count of the audio sequence, L CE Operators representing loss functions, in some embodiments, L CE It can be calculated using maximum likelihood estimation; s mix Represents a sequence of mixed speech; y i The text i in the target language corresponds to the audio i.

[0063] Similarly, the second loss function can be calculated using the following formula (3):

[0064] L j = L CE (s mix , y j(3)

[0065] Where j represents the count of the audio sequence, L CE Operators representing loss functions, in some embodiments, L CE It can be calculated using maximum likelihood estimation; s mix Represents a sequence of mixed speech; y j This represents the text j in the target language corresponding to the audio j.

[0066] The third loss function can be calculated using the following formula (4):

[0067] L MIX (s, y) = p*L i + (1 - p*)L j (4)

[0068] Among them, L MIX (s,y) represents the loss function for mixed speech features; p* represents the weights.

[0069] Therefore, the final loss function can be expressed as formula (5):

[0070] L1(s, y) = L CE (s, y) + L MIX (s, y) (5)

[0071] Where L1(s,y) represents the final loss function, L CE (s,y) represents L i With L j The sum of, L MIX (s,y) represents the loss function for mixed speech features.

[0072] By designing such a loss function, the speech translation model can output speech in the source language that is semantically similar to the target language text, and vice versa. Furthermore, the use of multi-level augmented data improves the robustness and accuracy of the translation model. Although Figure 3 and Figure 4 The examples illustrate three-level hybrid data augmentation at the word, sentence, and frame levels; however, embodiments of this disclosure can also be implemented in combination with two-level hybrid data augmentation or hybrid data augmentation at four or more levels.

[0073] Process 400 allows for the pre-training of the speech translation model (also known as the first stage). In some embodiments, the speech translation model can also be fine-tuned to achieve better translation results. Figure 5A schematic diagram of a process 500 for fine-tuning a speech translation model according to some embodiments of the present disclosure is shown.

[0074] The main principle of process 500 (also known as the second stage) is that the speech translation model 410 receives two types of data: one type is speech translation data pairs, which can consist of an audio clip of the source language and the corresponding text in the target language; the other type is machine translation data pairs, which consist of text clips of the source language and the corresponding text in the target language. In the first stage of fine-tuning, the input data consists only of speech translation data pairs. In the second stage of fine-tuning, the input data includes both speech translation data pairs and machine translation data pairs. Both types of data pairs are processed by the speech translation model 410. The model uses Jensen-Shannon divergence regularization in the output of the translation decoder 412 to prevent the speech translation model 410 from overfitting, i.e., from learning error features. The translation decoder 412 includes a speech translation decoder and a text translation decoder.

[0075] In process 500, audio S4 will be used as an example to describe the process of fine-tuning the speech translation model 410, but process 500 may include using more audio to fine-tune the speech translation model 410. Audio S4: That's my wish is converted into a word vector matrix by acoustic encoder 404 and input into speech translation model 410. The English text X4 corresponding to the English audio S4 is encoded as a word embedding 502. Word embedding 502 is also a vector, which can be obtained using one-hot encoding, and word embedding 502 is input into speech translation model 410.

[0076] On the other hand, during fine-tuning, the German text Y4 corresponding to the English audio S4, "Das ist meinWunsch," can be prepared as the correct translation result. The speech translation model 410 can calculate the loss function 508 (also known as the fourth loss function) associated with the audio S4 based on Y4 and the prediction result. The speech translation model 410 can also calculate the loss function 510 (also known as the fifth loss function) associated with the text X4 based on Y4 and the prediction result. The speech translation model 410 can also calculate the loss function 512 (also known as the sixth loss function) associated with both Y4 and X4.

[0077] Calculate the sum of the fourth, fifth, and sixth loss functions, and fine-tune the parameters of the speech translation model 410 to minimize the sum of these three loss functions. These parameters are then defined as the parameters of the fine-tuned speech translation model 410.

[0078] As an example, the fourth loss function and the fifth loss function in the second stage can be calculated using formulas (2) and (3), and the sixth loss function can be calculated using formula (6):

[0079]

[0080] pθ(yi|y<i,h(x))

[0081] where h() represents the content representation output by the translation encoder; pθ(yi|y<i,h(s)) represents the probability distribution output by the decoder of speech translation; pθ(yi|y<i,h(x)) represents the probability distribution output by the decoder of text translation.

[0082] As an example, the overall loss function in the fine-tuning stage can be expressed as:

[0083] L2(s,y) = L CE (s,y) + L CE (x,y) + L JSD (s,x,y) (7)

[0084] The fine-tuning process of the speech translation model is completed through the above process. The process 500 for fine-tuning enhances the knowledge conversion from machine translation to speech translation. Therefore, it further improves the generalization ability of the model and enhances the robustness of the model. And such data augmentation can be performed online without prior preparation. There is no need to perform data enhancement in advance before training, thus saving storage space.

[0085] In some embodiments, the trained speech translation model can be used to perform speech translation tasks. For example, receive an audio in the source language and generate the corresponding text in the target language based on the speech translation model.

[0086] Figure 6 FIG. shows a schematic diagram of an example architecture 600 of a speech translation model according to some embodiments of the present disclosure. As Figure 6 shown, the translation encoder 414 in the speech translation model 410 includes multiple layers. Only as an example, the translation encoder 414 has six layers, such as encoder layer 611, encoder layer 612, encoder layer 613, encoder layer 614, encoder layer 615, encoder layer 616, to fully extract the semantic information of the word vector 602 and encode its information matrix.

[0087] In one embodiment, a single encoder layer (e.g., encoder layer 611) may include two sub-layers. One layer is a multi-head attention layer that uses the attention mechanism to learn the relationships within the source text. The other layer is a feed-forward layer, such as a fully-connected network, which generates and outputs the encoded information matrix through multiple layers of linear transformations and activation functions (e.g., ReLU function).

[0088] The translation decoder 412 in the speech translation model 410 during the decoding phase may include multiple layers, such as six decoder layers: decoder layer 621, decoder layer 622, decoder layer 623, decoder layer 624, decoder layer 625, and decoder layer 626. Each decoder layer can decode based on the encoded information matrix and the output of the previous decoder layer to predict the probability of the next word. Based on the probability of each word at each position, the combination of words with the highest probability at each position can be selected as the target text 604 of the target language output.

[0089] Because the speech translation model 410 at this time is based on multi-level augmented data and fine-tuning, it can achieve more accurate translation results when translating speech from the source language to text in the target language. Although Figure 6 Six encoder and / or decoder layers are shown, but embodiments of this disclosure may have fewer or more encoder and / or decoder layers.

[0090] Figure 7 A block diagram of an apparatus 700 for speech translation according to certain embodiments of the present disclosure is shown. Figure 7 As shown, the apparatus 700 includes a speech acquisition module configured to acquire multiple audio samples of a source language. The apparatus 700 also includes a multi-level amplification module configured to generate mixed speech features by multi-level amplification of the multiple audio samples, wherein the multi-level amplification includes at least two of the word level, sentence level, or frame level. The apparatus 700 also includes a text generation module configured to generate target language text corresponding to the multiple audio samples based on the mixed speech features. The apparatus 700 may also include other modules to implement the steps of the method 200 according to embodiments of this disclosure; for brevity, these will not be described further here.

[0091] It is understood that the apparatus 700 of this disclosure can achieve at least one of the many advantages achievable by the methods or processes described above. For example, it can augment the original training data in different dimensions, such that the augmented training data includes varying degrees of semantic or timbre mixing, thereby enabling the speech translation model to learn more hidden features and thus making fuller use of the limited training data, thereby improving translation accuracy. The robustness of the model is enhanced through its generalization ability. Furthermore, this data augmentation method can be performed online without prior preparation. Data augmentation is not required before training, thus saving storage space.

[0092] The speech translation performance according to some embodiments of this disclosure can be referred to Table 1. Table 1 shows the experimental results, presented as Bilingual Evaluation Substitution Score (BLEU). The general idea of ​​BLEU score is accuracy. Given a standard translation as a reference, the sentence generated by the neural network is a candidate sentence of length n, and m words appear in the reference among the candidates. m / n can be a univariate formula for calculating BLEU.

[0093] Table 1 Comparison of BLEU scores

[0094] No machine translation data 26.4 31.0 28.6 There is machine-translated data 29.3 32.4 29.9

[0095] Table 2 shows the experimental results of the baseline. Existing baselines can be referenced by averaging the BLEU scores of various models, such as the W2V2-Transformer model, etc. For simplicity, Table 2 only shows the experimental results of one of these models.

[0096] Table 2 shows the baseline BLEU scores (using the W2V2-Transformer model as an example).

[0097] No machine translation data 24.3 29.6 26.7 There is machine-translated data 27.2 30.2 27.8

[0098] As can be seen from Tables 1 and 2, the speech translation model proposed in this disclosure has a higher translation effect than the baseline level, indicating that the scheme of this disclosure can make fuller use of limited data to train a better speech translation model, thereby providing higher translation accuracy.

[0099] Figure 8 A block diagram of an electronic device 800 according to certain embodiments of the present disclosure is shown. Device 800 may be the device or apparatus described in the embodiments of the present disclosure. Figure 8 As shown, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of device 800. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in... Figure 8 As shown, device 800 may also include a coprocessor.

[0100] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0102] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0103] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0104] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0105] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0106] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0107] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0109] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0110] The following are some example implementations of this disclosure.

[0111] Example 1. A speech translation method, comprising:

[0112] Retrieve multiple audio files in the source language;

[0113] By multi-level amplification of the multiple audio samples, mixed speech features are generated, wherein the multi-level amplification includes at least two of the following: word level, sentence level, or frame level; and

[0114] Based on the mixed speech features, target language text corresponding to the multiple audios is generated.

[0115] Example 2. According to the method of Example 1, wherein the plurality of audios includes a first audio and a second audio, and generating mixed speech features by multi-level amplification of the plurality of audios includes:

[0116] Replace at least one noun in the first text corresponding to the first audio with a synonym of the at least one noun; and

[0117] Based on the first audio and the audio of the synonym, a first word-level augmented audio is generated.

[0118] Example 3. The method according to any one of Examples 1-2, wherein generating mixed speech features by multi-level amplification of the plurality of audio samples further includes:

[0119] Generate the second word-level augmented audio of the second audio; and

[0120] A first sentence-level amplified audio is generated by concatenating the first word-level amplified audio with the second word-level amplified audio, wherein the first audio has a different timbre than the second audio.

[0121] Example 4. The method according to any one of Examples 1-3, wherein the plurality of audios further includes a third audio and a fourth audio, and generating mixed speech features by multi-level amplification of the plurality of audios further includes:

[0122] Generate the second-sentence hierarchical augmented audio of the third and fourth audio; and

[0123] Based on predetermined weights, the speech features of the first sentence-level augmented audio and the second sentence-level augmented audio are weighted to generate the frame-level mixed speech features.

[0124] Example 5. The method according to any one of Examples 1-4, wherein generating mixed speech features by multi-level amplification of the plurality of audios further includes:

[0125] Determine the first loss function associated with the first sentence-level amplified audio;

[0126] Determine the second loss function associated with the second sentence-level amplified audio;

[0127] Based on the predetermined weights, the first loss function and the second loss function are weighted together to obtain a third loss function; and

[0128] The speech translation model is trained using the first loss function, the second loss function, and the third loss function.

[0129] Example 6. The method according to any one of Examples 1-5 further includes:

[0130] A speech translation model is trained based on multi-level augmented training data and a loss function; and

[0131] The trained speech translation model is fine-tuned to obtain a fine-tuned speech translation model.

[0132] Example 7. The method according to any one of Examples 1-6, wherein the training data comprises:

[0133] The plurality of audio; and

[0134] The annotated text corresponding to the sentence-level augmented audio.

[0135] Example 8. The method according to any one of Examples 1-7, wherein fine-tuning the trained speech translation model to obtain a fine-tuned speech translation model includes:

[0136] Obtain a speech translation dataset, which includes multiple audio recordings of the source language and multiple corresponding texts of the target language;

[0137] Obtain a machine translation dataset, wherein the machine translation dataset includes multiple texts in the source language and corresponding multiple texts in the target language; and

[0138] The speech translation model is fine-tuned based on the speech translation dataset and the machine translation dataset.

[0139] Example 9. The method according to any one of Examples 1-8, wherein fine-tuning the speech translation model based on the speech translation dataset and the machine translation dataset includes:

[0140] Determine the fourth loss function for the speech translation task;

[0141] Determine the fifth loss function for the machine translation task;

[0142] Determine a sixth loss function associated with both the speech translation task and the machine translation task; and

[0143] Based on the fourth loss function, the fifth loss function, and the sixth loss function, the parameters of the speech translation model are adjusted to minimize the combined value of the fourth loss function, the fifth loss function, and the sixth loss function.

[0144] Example 10. The method according to any one of Examples 1-9 further includes:

[0145] Obtain the target audio of the source language; and

[0146] The speech translation model is used to translate the target audio into target text in the target language.

[0147] Example 11. The method according to any one of Examples 1-10, wherein generating target language text corresponding to the plurality of audios based on the mixed speech features comprises:

[0148] Determine multiple probabilities of multiple words at each position in the target language text; and

[0149] The word with the highest probability at each position is selected as the corresponding word in the target language text.

[0150] Example 12. The method according to any one of Examples 1-11 further includes:

[0151] Based on the hybrid speech features, the audio mask region is determined;

[0152] Based on the masked region, the hidden units of the audio are determined; and

[0153] Based on the hidden unit, word vectors for the audio are generated.

[0154] Example 13. The method according to any one of Examples 1-12 further includes:

[0155] The dimension of the word vectors is reduced to the target dimension using a convolutional neural network.

[0156] Example 14. An apparatus for speech translation, comprising:

[0157] The voice acquisition module is configured to acquire multiple audio recordings of the source language;

[0158] A multi-level amplification module is configured to generate mixed speech features by multi-level amplification of the plurality of audio samples, wherein the multi-level amplification includes at least two of the following: word level, sentence level, or frame level; and

[0159] The text generation module is configured to generate target language text corresponding to the multiple audios based on the mixed speech features.

[0160] Example 15. The apparatus according to Example 14, wherein the multi-level amplification module comprises:

[0161] The first replacement module is configured to replace at least one noun in the first text corresponding to the first audio with a synonym of the at least one noun; and

[0162] The word-level amplification module is configured to generate a first word-level amplified audio based on the first audio and the audio of the synonym.

[0163] Example 16. The apparatus according to any one of Examples 14-15, wherein the multi-level amplification module further comprises:

[0164] The second amplification module is configured to generate a second word-level amplified audio of the second audio; and

[0165] The first sentence-level amplification module is configured to generate a first sentence-level amplified audio by splicing the first word-level amplified audio with the second word-level amplified audio, wherein the first audio has a different timbre than the second audio.

[0166] Example 17. The apparatus according to any one of Examples 14-16, wherein the multi-level amplification module further comprises:

[0167] The second-sentence hierarchical augmentation module is configured to generate the second-sentence hierarchical augmented audio of the third and fourth audio; and

[0168] The first weighting module is configured to weight the speech features of the first sentence-level augmented audio and the second sentence-level augmented audio based on predetermined weights to generate the frame-level mixed speech features.

[0169] Example 18. The apparatus according to any one of Examples 14-17, wherein the multi-level amplification module further comprises:

[0170] The first loss function determination module is configured to determine a first loss function associated with the first sentence-level amplified audio.

[0171] The second loss function determination module is configured to determine a second loss function associated with the second sentence-level amplified audio.

[0172] The third loss function determination module is configured to weight the first loss function and the second loss function based on the predetermined weights to obtain a third loss function; and

[0173] The first training module is configured to train a speech translation model using the first loss function, the second loss function, and the third loss function.

[0174] Example 19. The apparatus according to any one of Examples 14-18 further includes:

[0175] The speech translation model training module is configured to train the speech translation model based on multi-level augmented training data and a loss function; and

[0176] The fine-tuning module is configured to fine-tune the trained speech translation model to obtain a fine-tuned speech translation model.

[0177] Example 20. The apparatus according to any one of Examples 14-19, wherein the training data comprises:

[0178] The plurality of audio; and

[0179] The annotated text corresponding to the sentence-level amplified audio.

[0180] Example 21. The apparatus according to any one of Examples 14-20, wherein the fine-tuning module comprises:

[0181] The speech translation dataset acquisition module is configured to acquire a speech translation dataset, which includes multiple audio recordings of the source language and multiple corresponding text recordings of the target language.

[0182] A machine translation dataset acquisition module is configured to acquire a machine translation dataset, wherein the machine translation dataset includes multiple texts in the source language and corresponding multiple texts in the target language; and

[0183] The second fine-tuning module is configured to fine-tune the speech translation model based on the speech translation dataset and the machine translation dataset.

[0184] Example 22. The apparatus according to any one of Examples 14-21, wherein the second fine-tuning module comprises:

[0185] The fourth loss function determination module is configured to determine the fourth loss function for the speech translation task;

[0186] The fifth loss function determination module is configured to determine the fifth loss function for the machine translation task;

[0187] The sixth loss function determination module is configured to determine a sixth loss function associated with both the speech translation task and the machine translation task; and

[0188] The minimization module is configured to adjust the parameters of the speech translation model based on the fourth loss function, the fifth loss function, and the sixth loss function, so as to minimize the combined value of the fourth loss function, the fifth loss function, and the sixth loss function.

[0189] Example 23. The apparatus according to any one of Examples 14-22 is further configured as follows:

[0190] The second speech acquisition module is configured to acquire the target audio of the source language; and

[0191] The speech translation module is configured to use the speech translation model to translate the target audio into target text in the target language.

[0192] Example 24. The apparatus according to any one of Examples 14-23, wherein the text generation module further comprises:

[0193] A probability determination module is configured to determine multiple probabilities of multiple words at each position in the target language text; and

[0194] The selection module is configured to select the word with the highest probability at each position as the corresponding word in the target language text.

[0195] Example 25. The apparatus according to any one of Examples 14-24 further includes:

[0196] The mask determination module is configured to determine the mask region of the audio based on the mixed speech features;

[0197] A hidden unit determination module is configured to determine hidden units of the audio based on the mask region; and

[0198] The word vector generation module is configured to generate word vectors for the audio based on the hidden unit.

[0199] Example 26. The apparatus according to any one of Examples 14-25 further includes:

[0200] A convolutional neural network module is configured to reduce the dimension of the word vectors to the target dimension using a convolutional neural network.

[0201] Example 27. An electronic device comprising:

[0202] Processor; and

[0203] A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including:

[0204] Retrieve multiple audio files in the source language;

[0205] By multi-level amplification of the multiple audio samples, mixed speech features are generated, wherein the multi-level amplification includes at least two of the following: word level, sentence level, or frame level; and

[0206] Based on the mixed speech features, target language text corresponding to the multiple audios is generated.

[0207] Example 28. An electronic device according to Example 27, wherein the plurality of audios includes a first audio and a second audio, and generating mixed speech features by multi-level amplification of the plurality of audios includes:

[0208] Replace at least one noun in the first text corresponding to the first audio with a synonym of the at least one noun; and

[0209] Based on the first audio and the audio of the synonym, a first word-level augmented audio is generated.

[0210] Example 29. An electronic device according to any one of Examples 27-28, wherein generating mixed speech features by multi-level amplification of the plurality of audio frequencies further includes:

[0211] Generate the second word-level augmented audio of the second audio; and

[0212] A first sentence-level amplified audio is generated by concatenating the first word-level amplified audio with the second word-level amplified audio, wherein the first audio has a different timbre than the second audio.

[0213] Example 30. An electronic device according to any one of Examples 27-29, wherein the plurality of audios further includes a third audio and a fourth audio, and generating mixed speech features by multi-level amplification of the plurality of audios further includes:

[0214] Generate the second-sentence hierarchical augmented audio of the third and fourth audio; and

[0215] Based on predetermined weights, the speech features of the first sentence-level augmented audio and the second sentence-level augmented audio are weighted to generate the frame-level mixed speech features.

[0216] Example 31. An electronic device according to any one of Examples 27-30, wherein generating mixed speech features by multi-level amplification of the plurality of audio frequencies further includes:

[0217] Determine the first loss function associated with the first sentence-level amplified audio;

[0218] Determine the second loss function associated with the second sentence-level amplified audio;

[0219] Based on the predetermined weights, the first loss function and the second loss function are weighted together to obtain a third loss function; and

[0220] The speech translation model is trained using the first loss function, the second loss function, and the third loss function.

[0221] Example 32. The electronic device according to any one of Examples 27-31, wherein the operation further includes:

[0222] A speech translation model is trained based on multi-level augmented training data and a loss function; and

[0223] The trained speech translation model is fine-tuned to obtain a fine-tuned speech translation model.

[0224] Example 33. An electronic device according to any one of Examples 27-32, wherein the training data comprises:

[0225] The plurality of audio; and

[0226] The annotated text corresponding to the sentence-level amplified audio.

[0227] Example 34. An electronic device according to any one of Examples 27-33, wherein fine-tuning the trained speech translation model to obtain a fine-tuned speech translation model includes:

[0228] Obtain a speech translation dataset, which includes multiple audio recordings of the source language and multiple corresponding texts of the target language;

[0229] Obtain a machine translation dataset, wherein the machine translation dataset includes multiple texts in the source language and corresponding multiple texts in the target language; and

[0230] The speech translation model is fine-tuned based on the speech translation dataset and the machine translation dataset.

[0231] Example 35. An electronic device according to any one of Examples 27-34, wherein fine-tuning the speech translation model based on the speech translation dataset and the machine translation dataset comprises:

[0232] Determine the fourth loss function for the speech translation task;

[0233] Determine the fifth loss function for the machine translation task;

[0234] Determine a sixth loss function associated with both the speech translation task and the machine translation task; and

[0235] Based on the fourth loss function, the fifth loss function, and the sixth loss function, the parameters of the speech translation model are adjusted to minimize the combined value of the fourth loss function, the fifth loss function, and the sixth loss function.

[0236] Example 36. The electronic device according to any one of Examples 27-35, wherein the operation further includes:

[0237] Obtain the target audio of the source language; and

[0238] The speech translation model is used to translate the target audio into target text in the target language.

[0239] Example 37. An electronic device according to any one of Examples 27-36, wherein generating target language text corresponding to the plurality of audios based on the mixed speech features includes:

[0240] Determine multiple probabilities of multiple words at each position in the target language text; and

[0241] The word with the highest probability at each position is selected as the corresponding word in the target language text.

[0242] Example 38. The electronic device according to any one of Examples 27-37, wherein the operation further includes:

[0243] Based on the hybrid speech features, the audio mask region is determined;

[0244] Based on the masked region, the hidden units of the audio are determined; and

[0245] Based on the hidden unit, word vectors for the audio are generated.

[0246] Example 39. The electronic device according to any one of Examples 27-38, wherein the operation further includes:

[0247] The word vectors are reduced to the target dimension using a convolutional neural network.

[0248] Example 40. A computer-readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of Examples 1 to 13.

[0249] Example 41. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 13.

[0250] Although this disclosure has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for speech translation, comprising: Retrieve multiple audio files in the source language; By multi-level amplification of the multiple audio samples, mixed speech features are generated. The multi-level amplification includes at least two of the following: word level, sentence level, or frame level. Frame level amplification includes: amplifying the audio samples obtained from word level amplification or sentence level amplification; and Based on the hybrid speech features, target language text corresponding to the multiple audio recordings is generated; The plurality of audio files includes a first audio file and a second audio file, wherein the first audio file has a different timbre than the second audio file, and the mixed speech features are generated by multi-level amplification of the plurality of audio files, including: Generate a first-word-level augmented audio of the first audio and a second-word-level augmented audio of the second audio; and The first sentence-level amplified audio is generated by concatenating the first word-level amplified audio with the second word-level amplified audio.

2. The method according to claim 1, wherein generating the first word-level augmented audio of the first audio comprises: Replace at least one noun in the first text corresponding to the first audio with a synonym of the at least one noun; as well as Based on the first audio and the audio of the synonym, the first word-level augmented audio is generated.

3. The method of claim 1, wherein the plurality of audios further includes a third audio and a fourth audio, and generating mixed speech features by multi-level amplification of the plurality of audios further includes: Generate the second-sentence hierarchical augmented audio of the third and fourth audio; as well as Based on predetermined weights, the speech features of the first sentence-level augmented audio and the second sentence-level augmented audio are weighted to generate the frame-level mixed speech features.

4. The method according to claim 3, wherein generating mixed speech features by multi-level amplification of the plurality of audio samples further comprises: Determine the first loss function associated with the first sentence-level amplified audio; Determine the second loss function associated with the second sentence-level amplified audio; Based on the predetermined weights, the first loss function and the second loss function are weighted together to obtain a third loss function; and The speech translation model is trained using the first loss function, the second loss function, and the third loss function.

5. The method according to claim 1, further comprising: A speech translation model is trained based on multi-level augmented training data and a loss function. as well as The trained speech translation model is fine-tuned to obtain the fine-tuned speech translation model.

6. The method of claim 5, wherein the training data comprises: The multiple audio; as well as Annotated text corresponding to the sentence-level amplified audio.

7. The method of claim 5, wherein fine-tuning the trained speech translation model to obtain a fine-tuned speech translation model comprises: Obtain a speech translation dataset, which includes multiple audio recordings of the source language and multiple corresponding texts of the target language; Obtain a machine translation dataset, which includes multiple texts in the source language and multiple texts in the corresponding target language; as well as The speech translation model is fine-tuned based on the speech translation dataset and the machine translation dataset.

8. The method of claim 7, wherein fine-tuning the speech translation model based on the speech translation dataset and the machine translation dataset comprises: Determine the fourth loss function for the speech translation task; Determine the fifth loss function for the machine translation task; Determine a sixth loss function that is associated with both the speech translation task and the machine translation task; as well as Based on the fourth loss function, the fifth loss function, and the sixth loss function, the parameters of the speech translation model are adjusted to minimize the combined value of the fourth loss function, the fifth loss function, and the sixth loss function.

9. The method of claim 8, further comprising: Obtain the target audio of the source language; as well as The speech translation model is used to translate the target audio into target text in the target language.

10. The method of claim 1, wherein generating target language text corresponding to the plurality of audios based on the mixed speech features comprises: Determine multiple probabilities of multiple words at each position in the target language text; as well as The word with the highest probability at each position is selected as the corresponding word in the target language text.

11. The method according to claim 1, further comprising: Based on the hybrid speech features, the audio mask region is determined; Based on the masked area, the hidden units of the audio are determined; as well as Based on the hidden unit, word vectors for the audio are generated.

12. The method of claim 11, further comprising: The dimension of the word vectors is reduced to the target dimension using a convolutional neural network.

13. An apparatus for speech translation, comprising: The voice acquisition module is configured to acquire multiple audio recordings of the source language; A multi-level amplification module is configured to generate mixed speech features by multi-level amplification of the plurality of audio samples, wherein the multi-level amplification includes at least two of the following: word level, sentence level, or frame level. Frame level amplification includes: amplifying the audio samples obtained from word level amplification or sentence level amplification; and The text generation module is configured to generate target language text corresponding to the multiple audios based on the mixed speech features; The plurality of audios includes a first audio and a second audio, the first audio having a different timbre than the second audio, and the multi-level amplification module is further configured to: Generate a first-word-level augmented audio of the first audio and a second-word-level augmented audio of the second audio; and The first sentence-level amplified audio is generated by concatenating the first word-level amplified audio with the second word-level amplified audio.

14. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 12.

Images