A Block-based Self-attention Real-time End-to-end Speech Translation Method

Abstract

The present invention discloses a block-based self-attention real-time end-to-end speech translation method, and the steps are as follows: preprocess the recorded audio file training data, map the ID of each piece of speech, its stored path, and the corresponding target language text to construct two mapping files; respectively extract two acoustic features of the audio, namely the Mel filter bank and the Mel-frequency cepstral coefficients; use the training data to construct a target language dictionary for generating the target language text sequence during decoding; clean the training data and convert it into a format file required by the end-to-end speech translation model; initialize the end-to-end speech translation model and train it with data files in a specific format; in the inference stage, set the size of the block, and use the trained end-to-end speech translation model to dynamically encode the source speech to achieve real-time generation of the target language sentence. The present invention enables the model to have the ability to perform speech translation in real time, and improves the decoding speed of the model without degrading the model performance.

Description

Technical Field

[0001] The present invention relates to an end-to-end real-time speech translation method, specifically a block-based self-attention real-time end-to-end speech translation method. Background Art

[0002] Speech Translation, in a broad sense, refers to the process of translating the speech of one language into the speech or text of the corresponding target language. Generally, Speech Translation is used to represent the process of translating speech into the text of the corresponding target language, while Speech-to-speech Translation specifically refers to the process of translating speech into the speech of the corresponding target language. Speech Translation has a very wide range of application scenarios, such as subtitle generation, conference simultaneous interpretation, etc., and plays an important role in cross-language communication.

[0003] In the past, speech translation usually adopted a cascaded method, that is, first through a speech recognition system, the speech was recognized into a text sequence of the source language, and then through a machine translation system, the text sequence was translated into a text sequence of the target language. The cascaded method can only perform translation in a serial manner, with a slow speed. Therefore, an end-to-end speech translation method has been developed, that is, it is not necessary to first recognize the source language speech into the text of the source language, but directly translate it into the target language text. The existing end-to-end speech translation models based on the Transformer structure have been able to achieve relatively good translation effects, and more model structures have been continuously proposed to improve the existing methods.

[0004] In some specific scenarios, it is required that the speech translation model has good real-time translation ability and can synchronously output the target language text without obtaining the entire input sequence. However, due to the calculation method limitations of the traditional attention mechanism, the end-to-end speech translation model based on Transformer cannot achieve real-time speech translation. Therefore, how to improve the calculation of attention to adapt to the real-time speech translation task has received more and more attention from researchers.

[0005] In existing real-time speech translation solutions, many are inspired by other real-time decoding tasks, such as real-time speech recognition, and the method of real-time decoding is migrated to real-time speech translation. Among them, the chunk-based real-time decoding scheme has achieved good results in real-time speech recognition tasks, and people are also researching how to apply this method to real-time speech translation tasks. The chunk-based attention calculation can be simply understood as setting a chunk window of a fixed size and performing attention calculation within this chunk window each time. However, due to a relatively strong constraint in the speech recognition task, that is, the input and output are monotonically aligned, there is no such constraint in the speech translation task. Therefore, directly migrating the chunk-based real-time speech recognition method to the real-time speech translation task has poor effects. For some sequence outputs with long-distance dependencies, how to obtain long-distance auxiliary information to generate the output is a difficult point in real-time speech translation. Summary of the Invention

[0006] Aiming at the deficiencies in the prior art such as the difficulty in obtaining long-distance sequence information in real-time speech translation, the technical problem to be solved by the present invention is to provide a chunk-based self-attention real-time end-to-end speech translation method, which uses the speech features obtained by preprocessing to train a speech translation model, and then performs real-time speech translation based on the chunk-based attention mechanism, enabling the speech translation model to obtain high efficiency and good results.

[0007] To solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] The present invention provides a chunk-based self-attention real-time end-to-end speech translation method, including the following steps:

[0009] 1) Preprocess the recorded audio file training data, map the ID of each speech, its stored path, and the corresponding target language text, and construct two mapping files;

[0010] 2) Extract the acoustic features of the audio file, and extract two acoustic features of the audio's mel filter bank (FBANK) and mel frequency cepstral coefficients (MFCC) respectively;

[0011] 3) Use the training data to construct a target language dictionary for generating the target language text sequence during decoding;

[0012] 4) Clean the training data and convert it into a format file required by the end-to-end speech translation model;

[0013] 5) Initialize the end-to-end speech translation model and train it with data files in a specific format;

[0014] 6) Enter the inference stage, set the block size, and use the trained end-to-end speech translation model to dynamically encode the source speech to achieve real-time generation of the target language sentence.

[0015] In step 1), preprocess the recorded audio file training data, map the ID of each speech, its stored path, and the corresponding target language text, and construct two mapping files, specifically:

[0016] 101) Construct a mapping file from speech ID to its stored path to quickly find the audio file of the speech;

[0017] 102) Construct a mapping file from speech ID to its corresponding target language text to quickly find the target language text of the speech.

[0018] In step 2), extract the acoustic features of the audio file, and extract two acoustic features, namely the Mel filter bank (FBANK) and the Mel frequency cepstral coefficients (MFCC) of the audio, specifically:

[0019] 201) By means of pre-emphasis, increase the energy of high-frequency speech, so that the information of high-frequency speech is more prominent;

[0020] 202) Use the rectangular window type, the window length is T, and when extracting features, perform discrete Fourier transform on the speech within each window to obtain a frame of speech information, and move backward by a distance P to continue the calculation of the next frame;

[0021] 203) Use the Mel filter to convert the energy spectrum into the Mel spectrum, and then take the logarithm of the Mel spectrum to obtain the FBANK feature;

[0022] 204) Perform inverse discrete Fourier transform on each frame of FBANK to further filter information and extract more useful feature information, namely the Mel frequency cepstral coefficients MFCC.

[0023] In step 5), use the MFCC features of the processed training data to train the end-to-end speech translation model, specifically:

[0024] 501) Encode the MFCC features into vector representations, and then send them into the end-to-end speech translation model in sequence, and update the parameters of the model through the backpropagation algorithm;

[0025] 502) When calculating the attention weights, add a noise term to improve the robustness of the model; then multiply a normalization coefficient before the attention weights to avoid the phenomenon of gradient explosion or gradient disappearance when the end-to-end speech translation model is trained.

[0026] In step 6), enter the inference stage, set the block size, and use the trained end-to-end speech translation model to perform dynamic encoding and decoding on the source speech to achieve real-time generation of the target language sentence. Specifically:

[0027] 601) Set the block size. During decoding, calculate the attention weights for the memory vectors within each fixed-size block, and then obtain the corresponding context vector by weighted summation. The block size should not be set too small to obtain more context information.

[0028] 602) During decoding, to determine the position of the block, calculate the selection probability of the last memory vector of the block selected during the previous output. If the probability is greater than or equal to 0.5, use this vector as the last unit of the block and perform attention calculation within the block of the previous window size. If the probability is less than 0.5, scan the next memory vector and recursively calculate the selection probability until the position of the block is determined.

[0029] 603) If the selection probability of all vectors is less than 0.5 until the last memory vector, set the context vector to the zero vector.

[0030] 604) Obtain the output target language text through the trained decoder.

[0031] The present invention has the following beneficial effects and advantages:

[0032] 1. The present invention proposes a block-based self-attention real-time end-to-end speech translation method. By extracting the acoustic features of speech, training a speech translation model, and adopting a block-based attention calculation method during decoding, real-time decoding is achieved to obtain the corresponding target language text.

[0033] 2. Compared with the existing real-time speech translation solutions, the present invention calculates attention based on blocks, which can effectively improve the decoding speed of the model and reduce the computational complexity, achieving fast synchronous translation of speech.

[0034] 3. The method used in the present invention has strong transferability and can be easily deployed to a speech translation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a diagram for extracting acoustic features in the present invention;

[0036] Figure 2 It is a diagram for real-time generating the target language text by the block-based attention method in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0037] The present invention will be further described below in conjunction with the accompanying drawings of the specification.

[0038] In real-time speech translation tasks, due to the limitations of traditional attention calculation methods, it is difficult to apply them to speech translation tasks. By using the attention method of blocks, the present invention can obtain certain context information during the decoding of a real-time speech translation model, and achieve real-time translation, improving the translation speed and reducing latency.

[0039] The present invention provides a block-based self-attention real-time end-to-end speech translation method, including the following steps:

[0040] 1) Preprocess the recorded audio file training data, map the ID of each speech, its stored path, and the corresponding target language text, and construct two mapping files;

[0041] 2) Extract the acoustic features of the audio file, and extract two acoustic features, namely the mel filter bank (FBANK) and the mel-frequency cepstral coefficients (MFCC) of the audio respectively;

[0042] 3) Use the training data to construct a target language dictionary for generating the target language text sequence during decoding;

[0043] 4) Clean the training data and convert it into a format file required by the end-to-end speech translation model;

[0044] 5) Initialize the end-to-end speech translation model and train it with data files in a specific format;

[0045] 6) Enter the inference stage, set the block size, and use the trained end-to-end speech translation model to dynamically encode the source speech to achieve real-time generation of the target language sentence.

[0046] In step 1), when preprocessing the recorded audio file training data, map the ID of each speech, its stored path, and the corresponding target language text, and construct two mapping files. Specifically:

[0047] 101) Construct a mapping file from the speech ID to its stored path for quickly finding the audio file of the speech;

[0048] 102) Construct a mapping file from the speech ID to its corresponding target language text for quickly finding the target language text of the speech.

[0049] In step 2), when extracting the acoustic features of the audio file, extract two acoustic features, namely the mel filter bank (FBANK) and the mel-frequency cepstral coefficients (MFCC) of the audio respectively. As Figure 1 shown, specifically:

[0050] 201) By means of pre-emphasis, increase the energy of high-frequency speech, so that the information of high-frequency speech is more prominent;

[0051] 202) Use the rectangular window type with the window length size being T. During feature extraction, perform discrete Fourier transform on the speech within each window to obtain a frame of speech information, and then move backward by a distance P to continue the calculation of the next frame;

[0052] 203) Use the Mel filter to convert the energy spectrum into the Mel spectrum, and then take the logarithm of the Mel spectrum to obtain the FBANK feature;

[0053] 204) Perform inverse discrete Fourier transform on each frame of the FBANK to further filter the information and extract more useful feature information, namely the Mel-frequency cepstral coefficients MFCC.

[0054] In step 5), use the MFCC features of the processed training data to train the end-to-end speech translation model. Specifically:

[0055] 501) Encode the MFCC features into vector representations, and then sequentially feed them into the end-to-end speech translation model to update the model's parameters through the backpropagation algorithm;

[0056] 502) When calculating the attention weights, add a noise term to improve the robustness of the model; then multiply a normalization coefficient before the attention weights to avoid the phenomenon of gradient explosion or gradient disappearance during the training of the end-to-end speech translation model.

[0057] In step 6), enter the inference stage, set the block size, and use the trained end-to-end speech translation model to perform dynamic encoding and decoding on the source speech to achieve real-time generation of the target language sentence, as Figure 2 shown specifically as:

[0058] 601) Set the block size. During decoding, calculate the attention weights for the memory vectors within each fixed-size block, and then obtain the corresponding context vector by weighted summation. The block size should not be set too small to obtain more context information;

[0059] 602) During decoding, to determine the position of the block, it is necessary to calculate the selection probability of the last memory vector of the block selected when generating the previous output. If the probability is greater than or equal to 0.5, use this vector as the last unit of the block and perform attention calculation within the block with a window size one block earlier. If the probability is less than 0.5, scan the next memory vector to recursively calculate the selection probability until the position of the block is determined;

[0060] 603) If the selection probability of all vectors is less than 0.5 until the last memory vector, set the context vector to the zero vector;

[0061] 604) Obtain the output target language text through the trained decoder.

[0062] In this embodiment, the proposed method is verified through the English-German speech translation task. The open MuST-C speech dataset is used as the training data in the experiment. After preprocessing, the FBANK features of the audio data are extracted by the acoustic feature extraction module. Then, the inverse discrete Fourier transform is performed on each frame of the FBANK to further extract the MFCC acoustic features. Next, the German text data of the training set is used to construct a German dictionary through word segmentation. In the training stage, the model parameters are randomly initialized. The MFCC acoustic features are fed into the encoder and encoded into vector representations, and the obtained vector information is then passed to the decoder for decoding. In the inference stage, the block size is set to 8, and the trained English-German speech translation model is used to decode the test set. Compared with the existing end-to-end speech translation model based on the wait-k strategy, the real-time end-to-end speech translation model proposed in this solution achieves a BLEU value of 15.22 and obtains a lower average latency (AL) with a higher BLEU value.

[0063] Table 1

[0064] system BLEU AL / ms End-to-End Speech Translation Model Based on the Wait-k Strategy 15.06 2805 Block-Based Self-Attention Real-Time End-to-End Speech Translation Model 15.22 2767

[0065] Table 1 BLEU values of different models trained and tested on the MuST-C dataset. The present invention achieves an effect comparable to that of offline speech translation on the existing training set and has been greatly improved in terms of real-time performance. Through the block-based attention calculation method, certain context information can be obtained without having to obtain the entire speech input sequence, and then the target language text can be generated in real time. This method has strong transferability and can be easily deployed in real-time speech translation systems.

Claims

1. A block-based self-attention real-time end-to-end speech translation method, characterized in that It includes the following steps: 1) Preprocess the recorded audio file training data, map the ID of each speech, its stored path, and the corresponding target language text, and construct two mapping files; 2) Extract the acoustic features of the audio file, and extract two acoustic features, namely the Mel filter bank and the Mel-frequency cepstral coefficients of the audio respectively; 3) Use the training data to construct a target language dictionary for generating the target language text sequence during decoding; 4) Clean the training data and convert it into a format file required by the end-to-end speech translation model; 5) Initialize the end-to-end speech translation model and train it with data files in a specific format; 6) Enter the inference stage, set the block size, and use the trained end-to-end speech translation model to dynamically encode the source speech to achieve real-time generation of the target language sentence; In step 6), when entering the inference stage, set the block size, and use the trained end-to-end speech translation model to perform dynamic encoding and decoding on the source speech to achieve real-time generation of the target language sentence. Specifically: 601) Set the block size. During decoding, calculate the attention weights for the memory vectors within each fixed-size block, and then obtain the corresponding context vector by weighted summation. The block size should not be set too small to obtain more context information; 602) During decoding, to determine the position of the block, it is necessary to calculate the selection probability of the last memory vector of the block selected during the previous output. If the probability is greater than or equal to 0.5, use this vector as the last unit of the block and perform attention calculation within the block of the previous window size. If the probability is less than 0.5, scan the next memory vector and recursively calculate the selection probability until the position of the block is determined; 603) If the selection probability of all vectors is less than 0.5 until the last memory vector, set the context vector to a zero vector; 604) Obtain the output target language text through the trained decoder.

2. The block-based self-attention real-time end-to-end speech translation method according to claim 1, wherein: In step 1), preprocess the recorded audio file training data, map the ID of each speech, its stored path, and the corresponding target language text, and construct two mapping files. Specifically: 101) Construct a mapping file from the speech ID to its stored path for quickly finding the audio file of the speech; 102) Construct a mapping file from the speech ID to its corresponding target language text for quickly finding the target language text of the speech.

3. The block-based self-attention real-time end-to-end speech translation method according to claim 1, wherein: In step 2), extract the acoustic features of the audio file, and extract two acoustic features, namely the Mel filter bank and the Mel-frequency cepstral coefficients of the audio respectively. Specifically: 201) By pre-emphasizing, increase the energy of the high-frequency speech, so that the information of the high-frequency speech is more prominent; 202) Use the rectangular window type, and the window length is T. During feature extraction, perform discrete Fourier transform on the speech within each window to obtain a frame of speech information, and move backward by a distance P to continue the calculation of the next frame; 203) Use the Mel filter to convert the energy spectrum into the Mel spectrum, and then take the logarithm of the Mel spectrum to obtain the Mel filter bank feature; 204) Perform the inverse discrete Fourier transform on each frame of the Mel filter bank to further filter the information and extract more useful feature information, namely the Mel-frequency cepstral coefficients.

4. The block-based self-attention real-time end-to-end speech translation method according to claim 1, wherein: In step 5), the Mel-frequency cepstral coefficient features of the processed training data are used to train the end-to-end speech translation model. Specifically: 501) Encode the Mel-frequency cepstral coefficient features into vector representations, and then sequentially feed them into the end-to-end speech translation model to update the model's parameters through the backpropagation algorithm; 502) When calculating the attention weights, add a noise term to improve the robustness of the model; then multiply a normalization coefficient before the attention weights to avoid the phenomenon of gradient explosion or gradient disappearance during the training of the end-to-end speech translation model.

Images